Systems and Methods for Automatic Exposure in High Dynamic Range Video Capture Systems

ABSTRACT

The various implementations described herein include methods, devices, and systems for implementing high dynamic range and automatic exposure functions in a video system. In one aspect, a method is performed at a video camera device and includes, while operating in a non-high dynamic range (HDR) mode: capturing first video data of a scene with the image sensor; determining whether a minimum number of pixels of the first video data meets one or more first color intensity criteria; and in accordance with the determination that the minimum number of pixels of the first video data meets the one or more first color intensity criteria, switching operation from the non-HDR mode to an HDR mode.

PRIORITY AND RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/652,400, filed Feb. 24, 2022, which in turn is a continuation of U.S.patent application Ser. No. 17/307,958, filed May 4, 2021, now U.S. Pat.No. 11,297,234, issued on Apr. 5, 2022, which in turn is a continuationof U.S. patent application Ser. No. 16/787,450, filed Feb. 11, 2020, nowU.S. Pat. No. 11,050,936, issued on Jun. 29, 2021, which in turn is acontinuation of U.S. patent application Ser. No. 15/987,831, filed May23, 2018, now U.S. Pat. No. 10,560,629, issued on Feb. 11, 2020, whichclaims priority to U.S. Provisional Application No. 62/510,241, filedMay 23, 2017, the disclosures of which are incorporated herein byreference.

This application is related to U.S. patent application Ser. No.15/987,835, filed May 23, 2018, which is hereby incorporated byreference in its entirety.

TECHNICAL FIELD

This relates generally to video camera devices, including but notlimited to, auto exposure and high dynamic range in video cameradevices.

BACKGROUND

In photography, exposure relates to an amount of light per unit areareaching a photographic film or image sensor. A camera in an automaticexposure (AE) mode automatically calculates and adjusts exposuresettings.

High Dynamic Range (HDR) photo techniques are used to capture images inwhich the scene content has very bright and very low level lightregions, sometimes called a high dynamic range scene. A high dynamicrange scene presents problems for standard imaging systems because theexposure value (combination of exposure time and gain) is fixed for theentire frame.

SUMMARY

Accordingly, there is a need for systems and/or devices with moreefficient and accurate methods for implementing high dynamic range andautomatic exposure functions in a video system. Such systems, devices,and methods optionally complement or replace conventional systems,devices, and methods for high dynamic range and automatic exposurefunctions.

HDR mode yields better image quality in high dynamic range scenes with awide variety of lighting conditions. However, it comes at a price. Aswill be discussed in greater detail below, the amount of image data thatmust be captured is generally doubled in HDR mode and there isconsiderable compute power required to do the fusion, both of whichimpact power consumption and raise the thermal load on the video system.Thus, intelligently switching into and out of HDR mode automatically inorder to minimize power and thermal impact is desirable in many videosystems.

In one aspect, some implementations include a method that is performedat a video camera device having memory, one or more processors, and animage sensor. The method includes, while operating in a high dynamicrange mode: (1) capturing video data of a scene in a field of view ofthe image sensor, including: (a) capturing a first subset of the videodata with a first exposure time; and (b) capturing a second subset ofthe video data with a second exposure time, lower than the firstexposure time; (2) determining whether the first subset of the videodata meets one or more first predefined criteria; (3) determiningwhether the second subset of the video data meets one or more secondpredefined criteria; (4) in accordance with a determination that thefirst subset meets the one or more first predefined criteria or adetermination that the second subset meets the one or more secondpredefined criteria, switching operation from the HDR mode to a non-HDRmode.

In another aspect, some implementations include a method performed at avideo camera device having memory, one or more processors, and an imagesensor. The method includes: (1) while operating in a high dynamic rangemode, capturing video data of a scene in a field of view of the imagesensor, including: (a) capturing a first subset of the video data with afirst exposure time; and (b) capturing a second subset of the video datawith a second exposure time, lower than the first exposure time; (2)combining first video data of the first subset of video data with secondvideo data of the second subset of video data to generate an HDR frame;and (3) adjusting a duration of at least one of the first exposure timeand the second exposure time based on one or more parameters of thecaptured video data, thereby altering a ratio of the first exposure timeto the second exposure time.

In another aspect, some implementations include a non-transitorycomputer-readable storage medium storing one or more programs, the oneor more programs comprising instructions which, when executed by acamera device with one or more processors, memory, and an image sensor,cause the camera device to perform any of the methods described herein.

In another aspect, some implementations include a camera device with oneor more processors, memory, and an image sensor configured to performany of the methods described herein.

Thus, devices are provided with more efficient, effective, and accuratemethods for implementing high dynamic range and automatic exposurefunctions in a video system. Such systems, devices, and methodsoptionally complement or replace conventional systems, devices, andmethods for high dynamic range and automatic exposure functions.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

For a better understanding of the various described implementations,reference should be made to the Description of Implementations below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 is an example smart home environment in accordance with someimplementations.

FIG. 2A is a block diagram illustrating a representative networkarchitecture that includes a smart home network in accordance with someimplementations.

FIG. 2B is a representative operating environment in which a serversystem interacts with client devices and smart devices in accordancewith some implementations.

FIG. 2C illustrates representative system architecture for eventanalysis and categorization in accordance with some implementations.

FIG. 3 is a block diagram illustrating a representative server system inaccordance with some implementations.

FIG. 4 is a block diagram illustrating a representative smart device inaccordance with some implementations.

FIGS. 5A-5B are block diagrams illustrating representative cameradevices in accordance with some implementations.

FIG. 6 is a block diagram illustrating a representative high dynamicrange mode in accordance with some implementations.

FIGS. 7A-7B are example long and short exposure images in accordancewith some implementations.

FIGS. 8A-8B are example light intensity histograms for the images ofFIGS. 7A-7B in accordance with some implementations.

FIG. 9 is a flow diagram illustrating a representative method ofdisabling a high dynamic range mode in a camera device in accordancewith some implementations.

FIG. 10 is a flow diagram illustrating another representative method ofdisabling a high dynamic range mode in a camera device in accordancewith some implementations.

FIG. 11 is a flow diagram illustrating another representative method ofdisabling a high dynamic range mode in a camera device in accordancewith some implementations.

FIG. 12 is a flow diagram illustrating another representative method ofdisabling a high dynamic range mode in a camera device in accordancewith some implementations.

FIG. 13 is a flow diagram illustrating a representative method ofenabling a high dynamic range mode in a camera device in accordance withsome implementations.

FIGS. 14A-14B are flow diagrams illustrating a representative method ofautomatically adjusting exposure in a camera device in accordance withsome implementations.

Like reference numerals refer to corresponding parts throughout theseveral views of the drawings.

DETAILED DESCRIPTION

Operating a camera device in an HDR mode has some drawbacks. Forexample, the sensor needs to deliver image data at twice the recordeddata rate, which increases power consumption. Increased powerconsumption may result in thermal dissipation challenges for the imagesensor since image quality performance tends to degrade as the sensor'stemperature increases. Thus, it is desirable to operate in an HDR modeonly when a high dynamic range scene is present so as to reduce powerconsumption and thermal heating.

In some implementations, a security camera captures surveillance videousing a digital imaging system. In some implementations, the digitalimages (also sometimes called frames) are captured as a sequence at aparticular frame rate. In some implementations, the images are thencompressed and sent to a server system (e.g., the “cloud”) for storageand retrieval. Each frame is composed of X by Y pixels depending on theresolution and, in some implementations, each pixel location has 3 colorcomponents: red, green and blue. In some implementations, framestatistics are gathered about each of the 3 color components (e.g., as acorresponding frame is captured). In some implementations, an averagelight intensity (“Luma” value) is calculated. In various situations andimplementations, a number of factors impact the average Luma value for aparticular frame, such as lighting conditions in the scene, distancefrom the camera to the scene, and reflectivity of objects within thescene. In some situations, these factors are part of the environment theuser wishes to monitor and cannot be controlled by user.

In some implementations, there are a number of controls within thecamera system that have an impact on the average Luma value and can beadjusted to move the average Luma value toward a particular target. Forexample, the controls optionally include one or more of: shutter speed,analog gain, digital gain, and frame rate. In some implementations, thecontrols are adjusted as part of an automatic exposure (AE) process inthe camera system.

In some implementations, a camera device utilizes an auto exposureprocess optimized for use with staggered HDR techniques. In someimplementations, the auto exposure process does one or more of thefollowing: uses fixed exposure ratio in order to simplify Local ToneMapping, determines optimal point in the pipeline at which to gatherstatistics; determines optimal scenarios to switch into and out of HDRmode; and calculates exposure, gain, and framerate for best imagequality using HDR.

Turning now to the figures, FIG. 1 is an example smart home environment100 in accordance with some implementations. The smart home environment100 includes a structure 150 (e.g., a house, office building, garage, ormobile home) with various integrated devices. It will be appreciatedthat devices may also be integrated into a smart home environment 100that does not include an entire structure 150, such as an apartment,condominium, or office space. Further, the smart home environment 100may control and/or be coupled to devices outside of the actual structure150. Indeed, several devices in the smart home environment 100 need notbe physically within the structure 150. For example, a devicecontrolling a pool heater 114 or irrigation system 116 may be locatedoutside of the structure 150.

It is to be appreciated that “smart home environments” may refer tosmart environments for homes such as a single-family house, but thescope of the present teachings is not so limited. The present teachingsare also applicable, without limitation, to duplexes, townhomes,multi-unit apartment buildings, hotels, retail stores, office buildings,industrial buildings, and more generally any living space or workspace.

It is also to be appreciated that while the terms user, customer,installer, homeowner, occupant, guest, tenant, landlord, repair person,and the like may be used to refer to the person or persons acting in thecontext of some particularly situations described herein, thesereferences do not limit the scope of the present teachings with respectto the person or persons who are performing such actions. Thus, forexample, the terms user, customer, purchaser, installer, subscriber, andhomeowner may often refer to the same person in the case of asingle-family residential dwelling, because the head of the household isoften the person who makes the purchasing decision, buys the unit, andinstalls and configures the unit, and is also one of the users of theunit. However, in other scenarios, such as a landlord-tenantenvironment, the customer may be the landlord with respect to purchasingthe unit, the installer may be a local apartment supervisor, a firstuser may be the tenant, and a second user may again be the landlord withrespect to remote control functionality. Importantly, while the identityof the person performing the action may be germane to a particularadvantage provided by one or more of the implementations, such identityshould not be construed in the descriptions that follow as necessarilylimiting the scope of the present teachings to those particularindividuals having those particular identities.

The depicted structure 150 includes a plurality of rooms 152, separatedat least partly from each other via walls 154. The walls 154 may includeinterior walls or exterior walls. Each room may further include a floor156 and a ceiling 158. Devices may be mounted on, integrated with and/orsupported by a wall 154, floor 156 or ceiling 158.

In some implementations, the integrated devices of the smart homeenvironment 100 include intelligent, multi-sensing, network-connecteddevices that integrate seamlessly with each other in a smart homenetwork (e.g., 202 FIG. 2A) and/or with a central server or acloud-computing system to provide a variety of useful smart homefunctions. The smart home environment 100 may include one or moreintelligent, multi-sensing, network-connected thermostats 102(hereinafter referred to as “smart thermostats 102”), one or moreintelligent, network-connected, multi-sensing hazard detection units 104(hereinafter referred to as “smart hazard detectors 104”), one or moreintelligent, multi-sensing, network-connected entryway interface devices106 and 120 (hereinafter referred to as “smart doorbells 106” and “smartdoor locks 120”), and one or more intelligent, multi-sensing,network-connected alarm systems 122 (hereinafter referred to as “smartalarm systems 122”).

In some implementations, the one or more smart thermostats 102 detectambient climate characteristics (e.g., temperature and/or humidity) andcontrol a HVAC system 103 accordingly. For example, a respective smartthermostat 102 includes an ambient temperature sensor.

The one or more smart hazard detectors 104 may include thermal radiationsensors directed at respective heat sources (e.g., a stove, oven, otherappliances, a fireplace, etc.). For example, a smart hazard detector 104in a kitchen 153 includes a thermal radiation sensor directed at astove/oven 112. A thermal radiation sensor may determine the temperatureof the respective heat source (or a portion thereof) at which it isdirected and may provide corresponding blackbody radiation data asoutput.

The smart doorbell 106 and/or the smart door lock 120 detects a person'sapproach to or departure from a location (e.g., an outer door), controldoorbell/door locking functionality (e.g., receive user inputs from aportable electronic device 166 to actuate bolt of the smart door lock120), announce a person's approach or departure via audio or visualmeans, and/or control settings on a security system (e.g., to activateor deactivate the security system when occupants go and come). In someimplementations, the smart doorbell 106 and/or the smart lock 120 arebattery-powered (e.g., are not line-powered). In some implementations,the smart doorbell 106 includes some or all of the components andfeatures of the camera 118. In some implementations, the smart doorbell106 includes a camera 118. In some implementations, the smart doorbell106 includes a camera 118 that is embedded in the doorbell 106. In someimplementations, the smart doorbell 106 includes a camera that ismounted on or near the doorbell 106. In some implementations, the smartdoorbell 106 includes a camera 118 that is not mounted in, on, or nearthe doorbell 106, but is instead mounted in proximity to the doorbell106. In some implementations, the smart doorbell 106 includes two ormore cameras 118 (e.g., one camera facing the entryway, and anothercamera facing approaching visitors). In some implementations, the smartdoorbell 106 has a camera (also sometimes referred to herein as doorbellcamera 106) which is separate from a video camera 118. For the purposesof this disclosure, video-related references to doorbell 106 refer toone or more cameras associated with doorbell 106.

The smart alarm system 122 may detect the presence of an individualwithin close proximity (e.g., using built-in IR sensors), sound an alarm(e.g., through a built-in speaker, or by sending commands to one or moreexternal speakers), and send notifications to entities or userswithin/outside of the smart home network 100. In some implementations,the smart alarm system 122 also includes one or more input devices orsensors (e.g., keypad, biometric scanner, NFC transceiver, microphone)for verifying the identity of a user, and one or more output devices(e.g., display, speaker). In some implementations, the smart alarmsystem 122 may also be set to an “armed” mode, such that detection of atrigger condition or event causes the alarm to be sounded unless adisarming action is performed.

In some implementations, the smart home environment 100 includes one ormore intelligent, multi-sensing, network-connected wall switches 108(hereinafter referred to as “smart wall switches 108”), along with oneor more intelligent, multi-sensing, network-connected wall pluginterfaces 110 (hereinafter referred to as “smart wall plugs 110”). Thesmart wall switches 108 detect ambient lighting conditions, detectroom-occupancy states, and/or control a power and/or dim state of one ormore lights. In some instances, smart wall switches 108 also control apower state or speed of a fan, such as a ceiling fan. The smart wallplugs 110 may detect occupancy of a room or enclosure and control supplyof power to one or more wall plugs (e.g., such that power is notsupplied to the plug if nobody is at home).

In some implementations, the smart home environment 100 of FIG. 1includes a plurality of intelligent, multi-sensing, network-connectedappliances 112 (hereinafter referred to as “smart appliances 112”), suchas refrigerators, stoves, ovens, televisions, washers, dryers, lights,stereos, intercom systems, garage-door openers, floor fans, ceilingfans, wall air conditioners, pool heaters, irrigation systems, securitysystems, space heaters, window AC units, motorized duct vents, and soforth. In some implementations, when plugged in, an appliance mayannounce itself to the smart home network, such as by indicating whattype of appliance it is, and it may automatically integrate with thecontrols of the smart home. Such communication by the appliance to thesmart home may be facilitated by either a wired or wirelesscommunication protocol. The smart home may also include a variety ofnon-communicating legacy appliances 140, such as old conventionalwasher/dryers, refrigerators, and the like, which may be controlled bysmart wall plugs 110. The smart home environment 100 may further includea variety of partially communicating legacy appliances 142, such asinfrared (“IR”) controlled wall air conditioners or other IR-controlleddevices, which may be controlled by IR signals provided by the smarthazard detectors 104 or the smart wall switches 108.

In some implementations, the smart home environment 100 includes one ormore network-connected cameras 118 that are configured to provide videomonitoring and security in the smart home environment 100. In someimplementations, the cameras 118 are battery-powered (e.g., are notline-powered). In some implementations, as described in more detailbelow, the cameras 118 are configured to selectively couple to one ormore networks and/or selectively capture, store, transmit video data(e.g., based on presence and characterization of motion within the fieldof view). In some implementations, in the low power mode, a camera 118detects an approaching visitor using a low power sensor, such as a PIRsensor, which is always on or periodically on.

In some implementations, the cameras 118 are used to determine occupancyof the structure 150 and/or particular rooms 152 in the structure 150,and thus act as occupancy sensors. For example, video captured by thecameras 118 may be processed to identify the presence of an occupant inthe structure 150 (e.g., in a particular room 152). Specific individualsmay be identified based, for example, on their appearance (e.g., height,face) and/or movement (e.g., their walk/gait). Cameras 118 mayadditionally include one or more sensors (e.g., IR sensors, motiondetectors), input devices (e.g., microphone for capturing audio), andoutput devices (e.g., speaker for outputting audio). In someimplementations, the cameras 118 are each configured to operate in a daymode and in a low-light mode (e.g., a night mode). In someimplementations, the cameras 118 each include one or more IRilluminators for providing illumination while the camera is operating inthe low-light mode. In some implementations, the cameras 118 include oneor more outdoor cameras. In some implementations, the outdoor camerasinclude additional features and/or components such as weatherproofingand/or solar ray compensation.

In some implementations, the smart home environment 100 includes one ormore network-connected doorbells 106 that are configured to providevideo monitoring and security in a vicinity of an entryway of the smarthome environment 100. The doorbells 106 are optionally used to determinethe approach and/or presence of a visitor. Specific individuals areoptionally identified based, for example, on their appearance (e.g.,height, face) and/or movement (e.g., their walk/gait). A doorbell 106optionally includes one or more sensors (e.g., IR sensors, motiondetectors), input devices (e.g., microphone for capturing audio), andoutput devices (e.g., speaker for outputting audio). In someimplementations, a doorbell 106 is configured to operate in a high-lightmode (e.g., a day mode) and in a low-light mode (e.g., a night mode). Insome implementations, a doorbell 106 includes one or more IRilluminators for providing illumination while the camera is operating inthe low-light mode. In some implementations, a doorbell 106 includes oneor more lights (e.g., one or more LEDs) for illuminating the doorbell inlow-light conditions and/or giving visual feedback to a visitor. In someimplementations, a doorbell 106 includes additional features and/orcomponents such as weatherproofing and/or solar ray compensation. Insome implementations, doorbell 106 is battery powered and runs in a lowpower or a high power mode. In some implementations, in the low powermode, doorbell 106 detects an approaching visitor using a low powersensor such as a PIR sensor which is always on or periodically on. Insome implementations, after the visitor approach is detected, doorbell106 switches to the high power mode to carry out further processingfunctions (described below).

In some implementations, the smart home environment 100 additionally oralternatively includes one or more other occupancy sensors (e.g., thesmart doorbell 106, smart door locks 120, touch screens, IR sensors,microphones, ambient light sensors, motion detectors, smart nightlights170, etc.). In some implementations, the smart home environment 100includes radio-frequency identification (RFID) readers (e.g., in eachroom 152 or a portion thereof) that determine occupancy based on RFIDtags located on or embedded in occupants. For example, RFID readers maybe integrated into the smart hazard detectors 104.

In some implementations, the smart home environment 100 includes one ormore devices outside of the physical home but within a proximategeographical range of the home. For example, the smart home environment100 may include a pool heater monitor 114 that communicates a currentpool temperature to other devices within the smart home environment 100and/or receives commands for controlling the pool temperature.Similarly, the smart home environment 100 may include an irrigationmonitor 116 that communicates information regarding irrigation systemswithin the smart home environment 100 and/or receives controlinformation for controlling such irrigation systems.

By virtue of network connectivity, one or more of the smart home devicesof FIG. 1 may further allow a user to interact with the device even ifthe user is not proximate to the device. For example, a user maycommunicate with a device using a computer (e.g., a desktop computer,laptop computer, or tablet) or other portable electronic device 166(e.g., a mobile phone, such as a smart phone). A webpage or applicationmay be configured to receive communications from the user and controlthe device based on the communications and/or to present informationabout the device's operation to the user. For example, the user may viewa current set point temperature for a device (e.g., a stove) and adjustit using a computer. The user may be in the structure during this remotecommunication or outside the structure.

As discussed above, users may control smart devices in the smart homeenvironment 100 using a network-connected computer or portableelectronic device 166. In some examples, some or all of the occupants(e.g., individuals who live in the home) may register their device 166with the smart home environment 100. Such registration may be made at acentral server to authenticate the occupant and/or the device as beingassociated with the home and to give permission to the occupant to usethe device to control the smart devices in the home. An occupant may usetheir registered device 166 to remotely control the smart devices of thehome, such as when the occupant is at work or on vacation. The occupantmay also use their registered device to control the smart devices whenthe occupant is actually located inside the home, such as when theoccupant is sitting on a couch inside the home. It should be appreciatedthat instead of or in addition to registering devices 166, the smarthome environment 100 may make inferences about which individuals live inthe home and are therefore occupants and which devices 166 areassociated with those individuals. As such, the smart home environmentmay “learn” who is an occupant and permit the devices 166 associatedwith those individuals to control the smart devices of the home.

In some implementations, in addition to containing processing andsensing capabilities, the devices 102, 104, 106, 108, 110, 112, 114,116, 118, 120, and/or 122 (collectively referred to as “the smartdevices”) are capable of data communications and information sharingwith other smart devices, a central server or cloud-computing system,and/or other devices that are network-connected. Data communications maybe carried out using any of a variety of custom or standard wirelessprotocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave,Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, etc.) and/or any of avariety of custom or standard wired protocols (e.g., Ethernet, HomePlug,etc.), or any other suitable communication protocol, includingcommunication protocols not yet developed as of the filing date of thisdocument.

In some implementations, the smart devices serve as wireless or wiredrepeaters. In some implementations, a first one of the smart devicescommunicates with a second one of the smart devices via a wirelessrouter. The smart devices may further communicate with each other via aconnection (e.g., network interface 160) to a network, such as theInternet 162. Through the Internet 162, the smart devices maycommunicate with a server system 164 (also called a central serversystem and/or a cloud-computing system herein). The server system 164may be associated with a manufacturer, support entity, or serviceprovider associated with the smart device(s). In some implementations, auser is able to contact customer support using a smart device itselfrather than needing to use other communication means, such as atelephone or Internet-connected computer. In some implementations,software updates are automatically sent from the server system 164 tosmart devices (e.g., when available, when purchased, or at routineintervals).

In some implementations, the network interface 160 includes aconventional network device (e.g., a router), and the smart homeenvironment 100 of FIG. 1 includes a hub device 180 that iscommunicatively coupled to the network(s) 162 directly or via thenetwork interface 160. The hub device 180 is further communicativelycoupled to one or more of the above intelligent, multi-sensing,network-connected devices (e.g., smart devices of the smart homeenvironment 100). Each of these smart devices optionally communicateswith the hub device 180 using one or more radio communication networksavailable at least in the smart home environment 100 (e.g., ZigBee,Z-Wave, Insteon, Bluetooth, Wi-Fi and other radio communicationnetworks). In some implementations, the hub device 180 and devicescoupled with/to the hub device can be controlled and/or interacted withvia an application running on a smart phone, household controller,laptop, tablet computer, game console or similar electronic device. Insome implementations, a user of such controller application can viewstatus of the hub device or coupled smart devices, configure the hubdevice to interoperate with smart devices newly introduced to the homenetwork, commission new smart devices, and adjust or view settings ofconnected smart devices, etc. In some implementations the hub deviceextends capabilities of low capability smart device to matchcapabilities of the highly capable smart devices of the same type,integrates functionality of multiple different device types—even acrossdifferent communication protocols, and is configured to streamlineadding of new devices and commissioning of the hub device. In someimplementations, hub device 180 further comprises a local storage devicefor storing data related to, or output by, smart devices of smart homeenvironment 100. In some implementations, the data includes one or moreof: video data output by a camera device, metadata output by a smartdevice, settings information for a smart device, usage logs for a smartdevice, and the like.

In some implementations, smart home environment 100 includes a localstorage device 190 for storing data related to, or output by, smartdevices of smart home environment 100. In some implementations, the dataincludes one or more of: video data output by a camera device (e.g., acamera included with doorbell 106), metadata output by a smart device,settings information for a smart device, usage logs for a smart device,and the like. In some implementations, local storage device 190 iscommunicatively coupled to one or more smart devices via a smart homenetwork (e.g., smart home network 202, FIG. 2A). In someimplementations, local storage device 190 is selectively coupled to oneor more smart devices via a wired and/or wireless communication network.In some implementations, local storage device 190 is used to store videodata when external network conditions are poor. For example, localstorage device 190 is used when an encoding bitrate of the cameraincluded with doorbell 106 exceeds the available bandwidth of theexternal network (e.g., network(s) 162). In some implementations, localstorage device 190 temporarily stores video data from one or moredoorbells (e.g., doorbell 106) prior to transferring the video data to aserver system (e.g., server system 164).

FIG. 2A is a block diagram illustrating a representative networkarchitecture 200 that includes a smart home network 202 in accordancewith some implementations. In some implementations, the smart devices204 in the smart home environment 100 (e.g., devices 102, 104, 106, 108,110, 112, 114, 116, 118, 120, and/or 122) combine with the hub device180 to create a mesh network in smart home network 202. In someimplementations, one or more smart devices 204 in the smart home network202 operate as a smart home controller. Additionally and/oralternatively, the hub device 180 operates as the smart home controller.In some implementations, a smart home controller has more computingpower than other smart devices. In some implementations, a smart homecontroller processes inputs (e.g., from smart devices 204, electronicdevice 166, and/or server system 164) and sends commands (e.g., to smartdevices 204 in the smart home network 202) to control operation of thesmart home environment 100. In some implementations, some of the smartdevices 204 in the smart home network 202 (e.g., in the mesh network)are “spokesman” nodes (e.g., 204-1) and others are “low-powered” nodes(e.g., 204-9). Some of the smart devices in the smart home environment100 are battery powered, while others have a regular and reliable powersource, such as by connecting to wiring (e.g., to 120V line voltagewires) behind the walls 154 of the smart home environment. The smartdevices that have a regular and reliable power source are referred to as“spokesman” nodes. These nodes are typically equipped with thecapability of using a wireless protocol to facilitate bidirectionalcommunication with a variety of other devices in the smart homeenvironment 100, as well as with the server system 164. In someimplementations, one or more “spokesman” nodes operate as a smart homecontroller. On the other hand, the devices that are battery powered arethe “low-power” nodes. These nodes tend to be smaller than spokesmannodes and typically only communicate using wireless protocols thatrequire very little power, such as Zigbee, ZWave, 6LoWPAN, Thread,Bluetooth, etc.

In some implementations, some low-power nodes are incapable ofbidirectional communication. These low-power nodes send messages, butthey are unable to “listen”. Thus, other devices in the smart homeenvironment 100, such as the spokesman nodes, cannot send information tothese low-power nodes.

In some implementations, some low-power nodes are capable of only alimited bidirectional communication. For example, other devices are ableto communicate with the low-power nodes only during a certain timeperiod.

As described, in some implementations, the smart devices serve aslow-power and spokesman nodes to create a mesh network in the smart homeenvironment 100. In some implementations, individual low-power nodes inthe smart home environment regularly send out messages regarding whatthey are sensing, and the other low-powered nodes in the smart homeenvironment—in addition to sending out their own messages—forward themessages, thereby causing the messages to travel from node to node(i.e., device to device) throughout the smart home network 202. In someimplementations, the spokesman nodes in the smart home network 202,which are able to communicate using a relatively high-powercommunication protocol, such as IEEE 802.11, are able to switch to arelatively low-power communication protocol, such as IEEE 802.15.4, toreceive these messages, translate the messages to other communicationprotocols, and send the translated messages to other spokesman nodesand/or the server system 164 (using, e.g., the relatively high-powercommunication protocol). Thus, the low-powered nodes using low-powercommunication protocols are able to send and/or receive messages acrossthe entire smart home network 202, as well as over the Internet 162 tothe server system 164. In some implementations, the mesh network enablesthe server system 164 to regularly receive data from most or all of thesmart devices in the home, make inferences based on the data, facilitatestate synchronization across devices within and outside of the smarthome network 202, and send commands to one or more of the smart devicesto perform tasks in the smart home environment.

As described, the spokesman nodes and some of the low-powered nodes arecapable of “listening.” Accordingly, users, other devices, and/or theserver system 164 may communicate control commands to the low-powerednodes. For example, a user may use the electronic device 166 (e.g., asmart phone) to send commands over the Internet to the server system164, which then relays the commands to one or more spokesman nodes inthe smart home network 202. The spokesman nodes may use a low-powerprotocol to communicate the commands to the low-power nodes throughoutthe smart home network 202, as well as to other spokesman nodes that didnot receive the commands directly from the server system 164.

In some implementations, a smart nightlight 170 (FIG. 1 ), which is anexample of a smart device 204, is a low-power node. In addition tohousing a light source, the smart nightlight 170 houses an occupancysensor, such as an ultrasonic or passive IR sensor, and an ambient lightsensor, such as a photo resistor or a single-pixel sensor that measureslight in the room. In some implementations, the smart nightlight 170 isconfigured to activate the light source when its ambient light sensordetects that the room is dark and when its occupancy sensor detects thatsomeone is in the room. In other implementations, the smart nightlight170 is simply configured to activate the light source when its ambientlight sensor detects that the room is dark. Further, in someimplementations, the smart nightlight 170 includes a low-power wirelesscommunication chip (e.g., a ZigBee chip) that regularly sends outmessages regarding the occupancy of the room and the amount of light inthe room, including instantaneous messages coincident with the occupancysensor detecting the presence of a person in the room. As mentionedabove, these messages may be sent wirelessly (e.g., using the meshnetwork) from node to node (i.e., smart device to smart device) withinthe smart home network 202 as well as over the Internet 162 to theserver system 164.

Other examples of low-power nodes include battery-powered versions ofthe smart hazard detectors 104, cameras 118, doorbells 106, and thelike. These battery-powered smart devices are often located in an areawithout access to constant and reliable power and optionally include anynumber and type of sensors, such as image sensor(s), occupancy/motionsensors, ambient light sensors, ambient temperature sensors, humiditysensors, smoke/fire/heat sensors (e.g., thermal radiation sensors),carbon monoxide/dioxide sensors, and the like. Furthermore,battery-powered smart devices may send messages that correspond to eachof the respective sensors to the other devices and/or the server system164, such as by using the mesh network as described above.

Examples of spokesman nodes include line-powered smart doorbells 106,smart thermostats 102, smart wall switches 108, and smart wall plugs110. These devices are located near, and connected to, a reliable powersource, and therefore may include more power-consuming components, suchas one or more communication chips capable of bidirectionalcommunication in a variety of protocols.

In some implementations, the smart home environment 100 includes servicerobots 168 (FIG. 1 ) that are configured to carry out, in an autonomousmanner, any of a variety of household tasks.

As explained above with reference to FIG. 1 , in some implementations,the smart home environment 100 of FIG. 1 includes a hub device 180 thatis communicatively coupled to the network(s) 162 directly or via thenetwork interface 160. The hub device 180 is further communicativelycoupled to one or more of the smart devices using a radio communicationnetwork that is available at least in the smart home environment 100.Communication protocols used by the radio communication network include,but are not limited to, ZigBee, Z-Wave, Insteon, EuOcean, Thread, OSIAN,Bluetooth Low Energy and the like. In some implementations, the hubdevice 180 not only converts the data received from each smart device tomeet the data format requirements of the network interface 160 or thenetwork(s) 162, but also converts information received from the networkinterface 160 or the network(s) 162 to meet the data format requirementsof the respective communication protocol associated with a targetedsmart device. In some implementations, in addition to data formatconversion, the hub device 180 further processes the data received fromthe smart devices or information received from the network interface 160or the network(s) 162 preliminary. For example, the hub device 180 canintegrate inputs from multiple sensors/connected devices (includingsensors/devices of the same and/or different types), perform higherlevel processing on those inputs—e.g., to assess the overall environmentand coordinate operation among the different sensors/devices—and/orprovide instructions to the different devices based on the collection ofinputs and programmed processing. It is also noted that in someimplementations, the network interface 160 and the hub device 180 areintegrated to one network device. Functionality described herein isrepresentative of particular implementations of smart devices, controlapplication(s) running on representative electronic device(s) (such as asmart phone), hub device(s) 180, and server(s) coupled to hub device(s)via the Internet or other Wide Area Network. All or a portion of thisfunctionality and associated operations can be performed by any elementsof the described system—for example, all or a portion of thefunctionality described herein as being performed by an implementationof the hub device can be performed, in different system implementations,in whole or in part on the server, one or more connected smart devicesand/or the control application, or different combinations thereof.

FIG. 2B illustrates a representative operating environment in which aserver system 164 provides data processing for monitoring andfacilitating review of events (e.g., motion, audio, security, etc.) fromdata captured by the smart devices 204, such as video cameras 118 ordoorbell cameras 106. As shown in FIG. 2B, the server system 164receives data from video sources 222 (including cameras 118 and/ordoorbell cameras 106) located at various physical locations (e.g.,inside or in proximity to homes, restaurants, stores, streets, parkinglots, and/or the smart home environments 100 of FIG. 1 ). In someimplementations, the video source(s) 222 are linked to more than onereviewer account (e.g., multiple user accounts may be subscribed to asingle smart home environment). In some implementations, the serversystem 164 provides video monitoring data for the video source 222 toclient devices 220 associated with the reviewer accounts. For example,the portable electronic device 166 is an example of the client device220. In some implementations, the server system 164 comprises a videoprocessing server that provides video processing services to the videosources and client devices 220. In some implementations, the serversystem 164 receives non-video data from one or more smart devices 204(e.g., audio data, metadata, numerical data, etc.). In someimplementations, the non-video data is analyzed to provide context formotion events detected by the video cameras 118 and/or doorbell cameras106. In some implementations, the non-video data indicates that an audioevent (e.g., detected by an audio device), security event (e.g.,detected by a perimeter monitoring device), hazard event (e.g., detectedby a hazard detector), medical event (e.g., detected by ahealth-monitoring device), or the like has occurred within a smart homeenvironment 100.

In some implementations, multiple reviewer accounts are linked to asingle smart home environment 100. For example, multiple occupants of asmart home environment 100 may have accounts liked to the smart homeenvironment. In some implementations, each reviewer account isassociated with a particular level of access. In some implementations,each reviewer account has personalized notification settings. In someimplementations, a single reviewer account is linked to multiple smarthome environments 100. For example, a person may own or occupy, or beassigned to review and/or govern, multiple smart home environments 100.In some implementations, the reviewer account has distinct levels ofaccess and/or notification settings for each smart home environment.

In some implementations, each of the video sources 222 includes one ormore video cameras 118 or doorbell cameras 106 that capture video andsend the captured video to the server system 164 substantially inreal-time. In some implementations, each of the video sources 222includes one or more doorbell cameras 106 that capture video and sendthe captured video to the server system 164 in real-time (e.g., within 1second, 10 seconds, 30 seconds, or 1 minute). In some implementations,each of the doorbells 106 include a video camera that captures video andsends the captured video to the server system 164 in real-time. In someimplementations, a video source 222 includes a controller device (notshown) that serves as an intermediary between the one or more doorbells106 and the server system 164. The controller device receives the videodata from the one or more doorbells 106, optionally performs somepreliminary processing on the video data, and sends the video dataand/or the results of the preliminary processing to the server system164 on behalf of the one or more doorbells 106 (e.g., in real-time). Insome implementations, each camera has its own on-board processingcapabilities to perform some preliminary processing on the capturedvideo data before sending the video data (e.g., along with metadataobtained through the preliminary processing) to the controller deviceand/or the server system 164. In some implementations, one or more ofthe cameras is configured to optionally locally store the video data(e.g., for later transmission if requested by a user). In someimplementations, a camera is configured to perform some processing ofthe captured video data, and, based on the processing, either send thevideo data in substantially real-time, store the video data locally, ordisregard the video data.

In accordance with some implementations, a client device 220 includes aclient-side module or smart home application, such as client-side module628 in FIG. 6 . In some implementations, the client-side modulecommunicates with a server-side module executed on the server system 164through the one or more networks 162. The client-side module providesclient-side functionality for the event monitoring and review processingand communications with the server-side module. The server-side moduleprovides server-side functionality for event monitoring and reviewprocessing for any number of client-side modules each residing on arespective client device 220. In some implementations, the server-sidemodule also provides server-side functionality for video processing andcamera control for any number of the video sources 222, including anynumber of control devices, cameras 118, and doorbells 106.

In some implementations, the server system 164 includes one or moreprocessors 212, a video storage database 210, an account database 214,an I/O interface to one or more client devices 216, and an I/O interfaceto one or more video sources 218. The I/O interface to one or moreclients 216 facilitates the client-facing input and output processing.The account database 214 stores a plurality of profiles for revieweraccounts registered with the video processing server, where a respectiveuser profile includes account credentials for a respective revieweraccount, and one or more video sources linked to the respective revieweraccount. The I/O interface to one or more video sources 218 facilitatescommunications with one or more video sources 222 (e.g., groups of oneor more doorbells 106, cameras 118, and associated controller devices).The video storage database 210 stores raw video data received from thevideo sources 222, as well as various types of metadata, such as motionevents, event categories, event category models, event filters, andevent masks, for use in data processing for event monitoring and reviewfor each reviewer account.

Examples of a representative client device 220 include a handheldcomputer, a wearable computing device, a personal digital assistant(PDA), a tablet computer, a laptop computer, a desktop computer, acellular telephone, a smart phone, an enhanced general packet radioservice (EGPRS) mobile phone, a media player, a navigation device, agame console, a television, a remote control, a point-of-sale (POS)terminal, a vehicle-mounted computer, an eBook reader, or a combinationof any two or more of these data processing devices or other dataprocessing devices.

Examples of the one or more networks 162 include local area networks(LAN) and wide area networks (WAN) such as the Internet. The one or morenetworks 162 are implemented using any known network protocol, includingvarious wired or wireless protocols, such as Ethernet, Universal SerialBus (USB), FIREWIRE, Long Term Evolution (LTE), Global System for MobileCommunications (GSM), Enhanced Data GSM Environment (EDGE), codedivision multiple access (CDMA), time division multiple access (TDMA),Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or anyother suitable communication protocol.

In some implementations, the server system 164 is implemented on one ormore standalone data processing apparatuses or a distributed network ofcomputers. In some implementations, the server system 164 also employsvarious virtual devices and/or services of third party service providers(e.g., third-party cloud service providers) to provide the underlyingcomputing resources and/or infrastructure resources of the server system164. In some implementations, the server system 164 includes, but is notlimited to, a server computer, a cloud server, a distributed cloudcomputing system, a handheld computer, a tablet computer, a laptopcomputer, a desktop computer, or a combination of any two or more ofthese data processing devices or other data processing devices.

In some implementations, a server-client environment includes both aclient-side portion (e.g., the client-side module) and a server-sideportion (e.g., the server-side module). The division of functionalitybetween the client and server portions of operating environment can varyin different implementations. Similarly, the division of functionalitybetween a video source 222 and the server system 164 can vary indifferent implementations. For example, in some implementations, theclient-side module is a thin-client that provides only user-facing inputand output processing functions, and delegates all other data processingfunctionality to a backend server (e.g., the server system 164).Similarly, in some implementations, a respective one of the videosources 222 is a simple video capturing device that continuouslycaptures and streams video data to the server system 164 with limited orno local preliminary processing on the video data. Although many aspectsof the present technology are described from the perspective of theserver system 164, the corresponding actions performed by a clientdevice 220 and/or the video sources 222 would be apparent to one ofskill in the art. Similarly, some aspects of the present technology maybe described from the perspective of a client device or a video source,and the corresponding actions performed by the video server would beapparent to one of skill in the art. Furthermore, some aspects may beperformed by the server system 164, a client device 220, and a videosource 222 cooperatively.

In some implementations, a video source 222 (e.g., a camera 118 ordoorbell 106 having an image sensor) transmits one or more streams ofvideo data to the server system 164. In some implementations, the one ormore streams include multiple streams, of respective resolutions and/orframe rates, of the raw video captured by the image sensor. In someimplementations, the multiple streams include a “primary” stream (e.g.,226-1) with a certain resolution and frame rate (e.g., corresponding tothe raw video captured by the image sensor), and one or more additionalstreams (e.g., 226-2 through 226-q). An additional stream is optionallythe same video stream as the “primary” stream but at a differentresolution and/or frame rate, or a stream that captures a portion of the“primary” stream (e.g., cropped to include a portion of the field ofview or pixels of the primary stream) at the same or differentresolution and/or frame rate as the “primary” stream. In someimplementations, the primary stream and/or the additional streams aredynamically encoded (e.g., based on network conditions, server operatingconditions, camera operating conditions, characterization of data in thestream (e.g., whether motion is present), user preferences, and thelike.

In some implementations, one or more of the streams 226 is sent from thevideo source 222 directly to a client device 220 (e.g., without beingrouted to, or processed by, the server system 164). In someimplementations, one or more of the streams is stored at the doorbell106 (e.g., in memory 406, FIG. 4 ) and/or a local storage device 190(e.g., a dedicated recording device), such as a digital video recorder(DVR). For example, in accordance with some implementations, thedoorbell 106 stores the most recent 24 hours of video footage recordedby the camera. As another example, in accordance with someimplementations, the doorbell 106 stores up to 24 hours of video footagerecorded by the camera (e.g., up to 24 hours of motion event data). Insome implementations, portions of the one or more streams are stored atthe doorbell 106 and/or the local storage device 109 (e.g., portionscorresponding to particular events or times of interest).

In some implementations, the server system 164 transmits one or morestreams of video data to a client device 220 to facilitate eventmonitoring by a user. In some implementations, the one or more streamsmay include multiple streams, of respective resolutions and/or framerates, of the same video feed. In some implementations, the multiplestreams include a “primary” stream with a certain resolution and framerate, corresponding to the video feed, and one or more additionalstreams. An additional stream may be the same video stream as the“primary” stream but at a different resolution and/or frame rate, or astream that shows a portion of the “primary” stream (e.g., cropped toinclude portion of the field of view or pixels of the primary stream) atthe same or different resolution and/or frame rate as the “primary”stream.

FIG. 2C illustrates a representative system architecture 240 includingvideo source(s) 241, server system 164, and client device(s) 220 inaccordance with some implementations. In some implementations, theserver system 164 includes functional modules for an event processor248, an event categorizer 252, an entity recognizer 250, and auser-facing frontend 254. The event processor 248 obtains the eventcandidates (e.g., by processing the video stream(s) 246 or by receivingevent start information from the video source 241, or by detecting auser press on a doorbell button of a doorbell camera). In someimplementations, the event candidates comprise motion event candidates.In some implementations, the event candidates comprise audio eventcandidates. In some implementations, the event candidates include a userpress on a doorbell button of a doorbell camera. In someimplementations, the event candidates include audio, electromagnetic,olfactory, and/or visual aspects. In some implementations, the eventcandidates include motion events, approach detections, and announcementdetections. The event categorizer 252 categorizes the event candidatesinto different event categories (e.g., based on data from the eventprocessor and/or the entity recognizer). The user-facing frontend 254generates event alerts and notifications, and facilitates review of thedetected entities and events by a reviewer through a review interface ona client device 220. The user-facing frontend 254 also receives useredits on the event and entity categories, user preferences for alertsand event filters, zone definitions for zones of interest, and the like.The event categorizer optionally revises event categorization models andresults based on the user edits received by the user-facing frontend.The entity recognizer optionally revises entity classifications and/orlabels based on the user edits received by the user-facing frontend. Insome implementations, the server system 164 also includes a video sourcedata database 256, person data 258, event categorization models database260, and event data and event masks database 262. In someimplementations, the person data 258 includes a person's database. Insome implementations, each of these databases is part of the serverdatabase 328 (e.g., part of data storage database 330).

The server system 164 receives one or more video stream(s) 246 from thevideo source 241 (e.g., a video source 222 from FIG. 2B) and optionallyreceives event candidate information 242, such as preliminarycharacterization information for detected entities and events (e.g.,entity and event metadata from processing performed at the doorbell106), and source information 244 such as device settings for a doorbell106. In some implementations, the event processor 248 communicates withthe video source 241 and/or one or more other devices of the smart homeenvironment, e.g., to request additional image data, audio data, andsensor data, such as high definition images or metadata for the videostream(s) 246. The server system sends alerts for events 264, alerts fordetected persons 266, event timeline information 268, and/or video data270 (e.g., still images or video clips corresponding to the detectedpersons and/or events) to the client device 220. In someimplementations, the alerts 264 distinguish visitor approach events fromother types of motion events. In some implementations, the alerts 264distinguish motion events captured at a doorbell 106 from motion eventscaptured by other smart devices (e.g., cameras 118). The server system164 optionally receives user information from the client device 220,such as event information 272 (e.g., edits to event categories), andzone definitions 274, and persons data 276 (e.g., classification ofdetected persons).

A data processing pipeline processes video information (e.g., a livevideo feed) received from a video source 241 (e.g., including a doorbell106 and an optional controller device) and/or audio information receivedfrom one or more smart devices in real-time (e.g., within 10 seconds, 30seconds, or 2 minutes) to identify and categorize events occurring inthe smart home environment, and sends real-time event alerts (e.g.,within 10 seconds, 20 seconds, or 30 seconds) and/or a refreshed eventtimeline (e.g., within 30 seconds, 1 minute, or 3 minutes) to a clientdevice 220 associated with a reviewer account for the smart homeenvironment. The data processing pipeline also processes storedinformation (such as stored video feeds from a video source 241) toreevaluate and/or re-categorize events as necessary, such as when newinformation is obtained regarding the event and/or when new informationis obtained regarding event categories (e.g., a new activity zonedefinition is obtained from the user).

After video and/or audio data is captured at a smart device, the data isprocessed to determine if any potential event candidates or persons arepresent. In some implementations, the data is initially processed at thesmart device (e.g., video source 241, camera 118, or doorbell 106).Thus, in some implementations, the smart device sends event candidateinformation, such as event start information, to the server system 164.In some implementations, the data is processed at the server system 164for event start detection. In some implementations, the video and/oraudio data is stored on server system 164 (e.g., in video sourcedatabase 256). In some implementations, the visual/audio data is storedon a server distinct from server system 164. In some implementations,after a motion start is detected, the relevant portion of the videostream is retrieved from storage (e.g., from video source database 256).

In some implementations, the event identification process includessegmenting the video stream into multiple segments then categorizing theevent candidate within each segment. In some implementations,categorizing the event candidate includes an aggregation of backgroundfactors, entity detection and identification, motion vector generationfor each motion entity, entity features, and scene features to generatemotion features for the event candidate. In some implementations, theevent identification process further includes categorizing each segment,generating or updating an event log based on categorization of asegment, generating an alert for the event based on categorization of asegment, categorizing the complete event, updating the event log basedon the complete event, and generating an alert for the event based onthe complete event. In some implementations, a categorization is basedon a determination that the event occurred within a particular zone ofinterest. In some implementations, a categorization is based on adetermination that the event candidate involves one or more zones ofinterest. In some implementations, a categorization is based on audiodata and/or audio event characterization.

The event analysis and categorization process may be performed by thesmart device (e.g., the video source 241) and the server system 164cooperatively, and the division of the tasks may vary in differentimplementations, for different equipment capability configurations,power parameters, and/or for different network, device, and server loadsituations. After the server system 164 categorizes the event candidate,the result of the event detection and categorization may be sent to areviewer associated with the smart home environment.

In some implementations, the server system 164 stores raw or compressedvideo data (e.g., in a video source database 256), event categorizationmodels (e.g., in an event categorization model database 260), and eventmasks and other event metadata (e.g., in an event data and event maskdatabase 262) for each of the video sources 241. In someimplementations, the video data is stored at one or more displayresolutions such as 480p, 780p, 1080i, 1080p, and the like.

In some implementations, the video source 241 (e.g., the doorbell 106)transmits a live video feed to the remote server system 164 via one ormore networks (e.g., the network(s) 162). In some implementations, thetransmission of the video data is continuous as the video data iscaptured by the doorbell 106. In some implementations, the transmissionof video data is irrespective of the content of the video data, and thevideo data is uploaded from the video source 241 to the server system164 for storage irrespective of whether any motion event has beencaptured in the video data. In some implementations, the video data isstored at a local storage device of the video source 241 by default, andonly video portions corresponding to motion event candidates detected inthe video stream are uploaded to the server system 164 (e.g., inreal-time or as requested by a user).

In some implementations, the video source 241 dynamically determines atwhat display resolution the video stream is to be uploaded to the serversystem 164. In some implementations, the video source 241 dynamicallydetermines which parts of the video stream are to be uploaded to theserver system 164. For example, in some implementations, depending onthe current server load and network conditions, the video source 241optionally prioritizes the uploading of video portions corresponding tonewly detected motion event candidates ahead of other portions of thevideo stream that do not contain any motion event candidates; or thevideo source 241 uploads the video portions corresponding to newlydetected motion event candidates at higher display resolutions than theother portions of the video stream. This upload prioritization helps toensure that important motion events are detected and alerted to thereviewer in real-time, even when the network conditions and server loadare less than optimal. In some implementations, the video source 241implements two parallel upload connections, one for uploading thecontinuous video stream captured by the doorbell 106, and the other foruploading video portions corresponding to detected motion eventcandidates. At any given time, the video source 241 determines whetherthe uploading of the continuous video stream needs to be suspendedtemporarily to ensure that sufficient bandwidth is given to theuploading of the video segments corresponding to newly detected motionevent candidates.

In some implementations, the video stream uploaded for cloud storage isat a lower quality (e.g., lower resolution, lower frame rate, highercompression, etc.) than the video segments uploaded for motion eventprocessing.

As shown in FIG. 2C, the video source 241 optionally includes a videodoorbell 106 and an optional controller device. In some implementations,the doorbell 106 includes sufficient on-board processing power toperform all necessary local video processing tasks (e.g., cuepointdetection for motion event candidates, video uploading prioritization,network connection management, etc.), and the doorbell 106 communicateswith the server system 164 directly, without any controller deviceacting as an intermediary. In some implementations, the doorbell 106captures the video data and sends the video data to the controllerdevice for the necessary local video processing tasks. The controllerdevice optionally performs the local processing tasks for multiplecameras. For example, there may be multiple cameras in one smart homeenvironment (e.g., the smart home environment 100, FIG. 1 ), and asingle controller device receives the video data from each camera andprocesses the video data to detect motion event candidates in the videostream from each camera. The controller device is responsible forallocating sufficient outgoing network bandwidth to transmitting videosegments containing motion event candidates from each camera to theserver before using the remaining bandwidth to transmit the video streamfrom each camera to the server system 164. In some implementations, thecontinuous video stream is sent and stored at one server facility whilethe video segments containing motion event candidates are send to andprocessed at a different server facility.

In some implementations, the smart device sends additional sourceinformation 503 to the server system 164. This additional sourceinformation 244 may include information regarding a device state (e.g.,IR mode, AE mode, DTPZ settings, etc.) and/or information regarding theenvironment in which the device is located (e.g., indoors, outdoors,night-time, day-time, etc.). In some implementations, the sourceinformation 244 is used by the server system 164 to perform eventdetection, entity recognition, and/or to categorize event candidates. Insome implementations, the additional source information 244 includes oneor more preliminary results from video processing performed by the videosource 241 (e.g., a doorbell 106), such as categorizations,object/entity recognitions, motion masks, and the like.

In some implementations, the video portion after an event start incidentis detected is divided into multiple segments. In some implementations,the segmentation continues until event end information (sometimes alsocalled an “end-of-event signal”) is obtained. In some implementations,the segmentation occurs within the server system 164 (e.g., by the eventprocessor 248). In some implementations, the segmentation comprisesgenerating overlapping segments. For example, a 10-second segment isgenerated every second, such that a new segment overlaps the priorsegment by 9 seconds.

In some implementations, each of the multiple segments is of the same orsimilar duration (e.g., each segment has a 10-12 second duration). Insome implementations, the first segment has a shorter duration than thesubsequent segments. Keeping the first segment short allows for realtime initial categorization and alerts based on processing the firstsegment. The initial categorization may then be revised based onprocessing of subsequent segments. In some implementations, a newsegment is generated if the motion entity enters a new zone of interest.

In some implementations, after the event processor module obtains thevideo portion corresponding to an event candidate, the event processor248 obtains background factors and performs motion entity detectionidentification, motion vector generation for each motion entity, andfeature identification. Once the event processor 248 completes thesetasks, the event categorizer 252 aggregates all of the information andgenerates a categorization for the motion event candidate. In someimplementations, the event processor 248 and the event categorizer 252are components of the video processing module 322 (FIG. 3 ). In someimplementations, false positive suppression is optionally performed toreject some motion event candidates before the motion event candidatesare submitted for event categorization. In some implementations,determining whether a motion event candidate is a false positiveincludes determining whether the motion event candidate occurred in aparticular zone. In some implementations, determining whether a motionevent candidate is a false positive includes analyzing an importancescore for the motion event candidate. The importance score for a motionevent candidate is optionally based on zones of interest involved withthe motion event candidate, background features, motion vectors, scenefeatures, entity features, motion features, motion tracks, and the like.

In some implementations, the video source 241 has sufficient processingcapabilities to perform, and does perform, entity detection, personrecognition, background estimation, motion entity identification, themotion vector generation, and/or the feature identification.

FIG. 3 is a block diagram illustrating the server system 164 inaccordance with some implementations. The server system 164 includes oneor more processing units (CPUs) 302, one or more network interfaces 304(e.g., including an I/O interface to one or more client devices and anI/O interface to one or more electronic devices), memory 306, and one ormore communication buses 308 for interconnecting these components(sometimes called a chipset). The memory 306 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR SRAM, or other random accesssolid state memory devices; and, optionally, includes non-volatilememory, such as one or more magnetic disk storage devices, one or moreoptical disk storage devices, one or more flash memory devices, or oneor more other non-volatile solid state storage devices. The memory 306,optionally, includes one or more storage devices remotely located fromone or more processing units 302. The memory 306, or alternatively thenon-volatile memory within memory 306, includes a non-transitorycomputer-readable storage medium. In some implementations, the memory306, or the non-transitory computer-readable storage medium of thememory 306, stores the following programs, modules, and data structures,or a subset or superset thereof:

-   -   an operating system 310 including procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 312 for connecting the server        system 164 to other systems and devices (e.g., client devices,        electronic devices, and systems connected to one or more        networks 162) via one or more network interfaces 304 (wired or        wireless);    -   a server-side module 314, which provides server-side        functionalities for device control, data processing, and data        review, including, but not limited to:        -   a data receiving module 316 for receiving data from            electronic devices (e.g., video data from a doorbell 106,            FIG. 1 ), and preparing the received data for further            processing and storage in the data storage database 3160;        -   a device control module 318 for generating and sending            server-initiated control commands to modify operation modes            of electronic devices (e.g., devices of a smart home            environment 100), and/or receiving (e.g., from client            devices 220) and forwarding user-initiated control commands            to modify operation modes of the electronic devices;        -   a data processing module 320 for processing the data            provided by the electronic devices, and/or preparing and            sending processed data to a device for review (e.g., client            devices 220 for review by a user), including, but not            limited to:            -   a video processor sub-module 322 for processing (e.g.,                categorizing and/or recognizing) detected entities                and/or event candidates within a received video stream                (e.g., a video stream from doorbell 106);            -   a user interface sub-module 324 for communicating with a                user (e.g., sending alerts, timeline events, etc. and                receiving user edits and zone definitions and the like);                and    -   a server database 328, including but not limited to:        -   a data storage database 330 for storing data associated with            each electronic device (e.g., each doorbell) of each user            account, as well as data processing models, processed data            results, and other relevant metadata (e.g., names of data            results, location of electronic device, creation time,            duration, settings of the electronic device, etc.)            associated with the data, where (optionally) all or a            portion of the data and/or processing associated with the            hub device 180 or smart devices are stored securely;        -   an account database 332 for storing account information for            user accounts, including user account information such as            user profiles 334, information and settings for linked hub            devices and electronic devices (e.g., hub device            identifications), hub device specific secrets, relevant user            and hardware characteristics (e.g., service tier, device            model, storage capacity, processing capabilities, etc.),            user interface settings, data review preferences, etc.,            where the information for associated electronic devices            includes, but is not limited to, one or more device            identifiers (e.g., MAC address and UUID), device specific            secrets, and displayed titles;        -   a device information database 336 for storing device            information related to one or more devices such as device            profiles 338, e.g., device identifiers and hub device            specific secrets, independently of whether the corresponding            hub devices have been associated with any user account;        -   an event information database 340 for storing event            information such as event records 342 and context            information, e.g., contextual data describing circumstances            surrounding an approaching visitor; and        -   characterization data 348 for use with characterizing            motion, persons, and events within the smart home            environment, e.g., in conjunction with data processing            module 320.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwiserearranged in various implementations. In some implementations, thememory 306, optionally, stores a subset of the modules and datastructures identified above. Furthermore, the memory 306, optionally,stores additional modules and data structures not described above (e.g.,an account management module for linking client devices, smart devices,and smart home environments).

FIG. 4 is a block diagram illustrating a representative smart device 204in accordance with some implementations. In some implementations, thesmart device 204 (e.g., any devices of a smart home environment 100,FIG. 1 ) includes one or more processing units (e.g., CPUs, ASICs,FPGAs, microprocessors, and the like) 402, one or more communicationinterfaces 404 with radios 406, user interface 410, image sensor(s) 418,sensor(s) 422, memory 426, and one or more communication buses 408 forinterconnecting these components (sometimes called a chipset). In someimplementations, the user interface 410 includes one or more outputdevices 412 that enable presentation of media content, including one ormore speakers and/or one or more visual displays. In someimplementations, the user interface 410 includes one or more inputdevices 414, including user interface components that facilitate userinput such as a keyboard, a mouse, a voice-command input unit ormicrophone, a touch screen display, a touch-sensitive input pad, agesture capturing camera, or other input buttons or controls. In someimplementations, an input device 414 for a doorbell 106 is a tactile ortouch-sensitive doorbell button. Furthermore, some smart devices 204 usea microphone and voice recognition or a camera and gesture recognitionto supplement or replace the keyboard.

The sensor(s) 422 include, for example, one or more thermal radiationsensors, ambient temperature sensors, humidity sensors, infrared (IR)sensors such as passive infrared (PIR) sensors, proximity sensors, rangesensors, occupancy sensors (e.g., using RFID sensors), ambient lightsensors (ALS), motion sensors 424, location sensors (e.g., GPS sensors),accelerometers, and/or gyroscopes.

The communication interfaces 404 include, for example, hardware capableof data communications using any of a variety of custom or standardwireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread,Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, etc.) and/or anyof a variety of custom or standard wired protocols (e.g., Ethernet,HomePlug, etc.), or any other suitable communication protocol, includingcommunication protocols not yet developed as of the filing date of thisdocument. The radios 406 enable one or more radio communication networksin the smart home environments, and enable a smart device 204 tocommunicate with other devices. In some implementations, the radios 406are capable of data communications using any of a variety of custom orstandard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee,6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi,etc.).

The memory 426 includes high-speed random access memory, such as DRAM,SRAM, DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. The memory 426, or alternatively the non-volatilememory within the memory 426, includes a non-transitorycomputer-readable storage medium. In some implementations, the memory426, or the non-transitory computer-readable storage medium of thememory 426, stores the following programs, modules, and data structures,or a subset or superset thereof:

-   -   operating logic 428 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   a communication module 430 for coupling to and communicating        with other network devices (e.g., a network interface 160, such        as a router that provides Internet connectivity, networked        storage devices, network routing devices, a server system 164,        other smart devices 204, client devices 220, etc.) connected to        one or more networks 162 via one or more communication        interfaces 404 (wired or wireless);    -   an input processing module 432 for detecting one or more user        inputs or interactions from the one or more input devices 414        and interpreting the detected inputs or interactions;    -   a user interface module 434 for providing and presenting a user        interface in which settings, captured data, and/or other data        for one or more devices (e.g., the smart device 204, and/or        other devices in a smart home environment 100) can be configured        and/or viewed;    -   one or more applications 436 for execution by the smart device        (e.g., games, social network applications, smart home        applications, and/or other web or non-web based applications)        for controlling devices (e.g., executing commands, sending        commands, and/or configuring settings of the smart device 204        and/or other client/electronic devices), and for reviewing data        captured by devices (e.g., device status and settings, captured        data, or other information regarding the smart device 204 and/or        other client/electronic devices);    -   a device-side module 438, which provides device-side        functionalities for device control, data processing and data        review, including but not limited to:        -   a command module 440 for receiving, forwarding, and/or            executing instructions and control commands (e.g., from a            client device 220, from a server system 164, from user            inputs detected on the user interface 410, etc.) for            operating the smart device 204; and        -   a data processing module 442 for processing data captured or            received by one or more inputs (e.g., input devices 414,            image sensor(s) 418, sensors 422, interfaces (e.g.,            communication interfaces 404, radios 440), and/or other            components of the smart device 204, and for preparing and            sending processed data to a remote device (e.g., client            devices 220) for review by a user;    -   a camera module 444 for operating the image sensor(s) 418 and        associated circuitry, e.g., for enabling and disabling the image        sensor(s) 418 based on data from one or more low-power sensors        422 (e.g., data from a PIR sensor or ALS), including but not        limited to:        -   an exposure module 446 for adjusting exposure parameters of            the image sensor(s) 418, such as shutter speed, gain,            exposure ratios, and light intensity parameters; and        -   a dynamic range module 448 for adjusting exposure times of            the image sensor(s) 418 (e.g., automatically switching into            and out of an HDR mode);    -   an event analysis module 450 for analyzing captured sensor data,        e.g., to detect and/or recognize approaching visitors and        context information; and    -   device data 548 storing data associated with devices (e.g., the        smart device 204), including, but is not limited to:        -   account data 460 storing information related to user            accounts linked to the smart device 204, e.g., including            cached login credentials, smart device identifiers (e.g.,            MAC addresses and UUIDs), user interface settings, display            preferences, authentication tokens and tags, password keys,            and the like;        -   local data storage 462 for selectively storing raw or            processed data associated with the smart device 204, such as            event data and/or video data captured by the image sensor(s)            418;        -   camera data 464 storing information related operation of the            image sensor, such as frame rate, shutter speed, analog            gain, digital gain, image encoding, and compression; and        -   characterization data 470 for entities, persons, and/or            events detected by, or associated with, the smart device 204            (e.g., data generated or used by the characterization module            456).

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwiserearranged in various implementations. In some implementations, thememory 426, optionally, stores a subset of the modules and datastructures identified above. Furthermore, the memory 426, optionally,stores additional modules and data structures not described above, suchas a sensor management module for managing operation of the sensor(s)422. In some implementations, one or more operations of the smart device204 are performed by the server system 164. These operations include,but are not necessarily limited to, operations performed by or undercontrol of computer program instructions such as the applications 436,device-side module 438, camera module 444 and event and event analysismodule 450. In some implementations, device data 458 associated withthese operations that are performed by the server system 164 are stored,maintained and updated in whole or in part on or by the server system164.

FIGS. 5A-5B are block diagrams illustrating representative cameradevices in accordance with some implementations. In someimplementations, the camera devices 501 and 521 comprise smart devices204. In some implementations, the memory 508 is the memory 426 and/orstores the programs, modules, and data structures of memory 426. In someimplementations, the sensor 504 is an image sensor 418 and/or operatesin a similar manner.

In various implementations, exposure statistics for HDR imaging aregathered in differing manners. For example, statistics are optionallygathered on: (1) only one of the two exposure times; (2) each of theexposure times separately; (3) both of the exposure times together; or(4) the fused image (e.g., before local tone mapping).

FIG. 5A shows the camera device 501 configured to obtain and storeexposure statistics prior to fusion of short and long exposure images inaccordance with option 2 above. As shown in FIG. 5A, image data iscaptured via the lens 502 at the image sensor 504. In an HDR mode, theimage data is captured at differing exposure times. In someimplementations, the image data is stored in memory 508 (e.g., a memorybuffer, cache, or other device). In some implementations, exposurestatistics are obtained for each frame and/or exposure time. In someimplementations, the exposure statistics include light intensity dataand/or color data for each pixel. The image data for the short and longexposure times is combined (fused) at a fusion module 510 (e.g., withina graphics processing unit (GPU)). In some implementations, the fusionmodule 510 is the camera module 444. Once the image data is combined,additional image processing, such as local tone mapping (LTM), isoptionally performed. In the example shown by FIG. 5A, exposurestatistics are gathered on the individual frames (e.g., the exposuredata is 10 bits wide as shown by the long exposure 602 and shortexposure 604 in FIG. 6 ).

FIG. 5B shows the camera device 521 configured to obtain and storeexposure statistics after fusion of short and long exposure images inaccordance with option 4 above. As with the camera device 501, in thecamera device 521 image data is captured via the lens 502 at the imagesensor 504, and, in an HDR mode, the image data is captured at differingexposure times. In some implementations, the image data is stored inmemory 508. The image data for the short and long exposure times iscombined (fused) at the fusion module 510 (e.g., within a graphicsprocessing unit (GPU)). Once the image data is combined, exposurestatistics are obtained for the fused image (e.g., light intensity dataand data indicating whether each pixel was selected from the shortexposure image or the long exposure image) and additional imageprocessing, such as LTM, is optionally performed. In the example shownby FIG. 5B, exposure statistics are gathered on the raw fused imageafter fusion (e.g., the exposure data is 13 bits wide as shown by thecomposite exposure 606 in FIG. 6 ).

HDR photo techniques are used to capture images in which the scenecontent has a very wide range of lighting conditions, specifically verybright and very low level light in the same scene (also sometimes calleda high dynamic range scene). A high dynamic range scene presentsproblems for standard imaging systems because the exposure value(combination of exposure time and gain) is fixed for the entire frame.Conventional HDR techniques capture multiple frame exposures then animage processor fuses parts from the different frames to create acomposite image. Optionally, the image processor applies Local ToneMapping (LTM) to the composite to smooth the colors at the fusionpoints. These techniques are generally not desirable for use in a videosystem since full frame exposures are separated by too much timeresulting in undesirable motion artifacts.

FIG. 6 is a block diagram illustrating a representative high dynamicrange mode in accordance with some implementations. As shown in FIG. 6 ,a long exposure image 602 is used to capture low-to-medium lux level(low light) portions of the scene, and a short exposure image 604 isused to capture medium-to-high lux level portions of the scene. The longexposure image 602 and the short exposure image 604 are combined tocreate a composite exposure image 606 capturing low-to-high lux levels.For example, the low lux level portions of the scene captured in theshort exposure image 604 are combined with the high lux level portionsof the scene captured in the long exposure image 602.

In some implementations, a staggered HDR process is utilized by thecamera device (e.g., the camera device 501 or 521). A staggered HDRprocess also involves multiple exposures, but instead of being performeda full frame at time, the multitude of exposures are delivered forportions of the frame, e.g., one “frame line” at a time. As used herein,a “frame line” is a line (e.g., a horizontal or vertical line) in aframe of video. For example, a sensor operating in a “2 frame staggeredHDR” exposes each line to two different exposure times, a long exposureand a short exposure, and it delivers each line of the frame twice,e.g., first the short exposure version then the long exposure version.In this example, line 1 of the frame would come out twice, then line twowould come out twice, then line three, and so on. In accordance withsome implementations, an image processor generates a composite of thetwo lines in order to create an optimal frame. In some implementations,an automatic exposure (AE) process is integrated with the HDR process,so as to determine optimal exposures to enable the creation of optimalcomposite frames.

FIGS. 7A-7B are example long and sure exposure images in accordance withsome implementations. FIG. 7A shows a long exposure image 702 and FIG.7B shows a short exposure image 704. In accordance with someimplementations, select pixels from the long exposure image 702 and theshort exposure image 704 are combined (fused) to form a composite imagewithout the over- or under-exposed regions.

FIGS. 8A-8B are example light intensity histograms for the images ofFIGS. 7A-7B in accordance with some implementations. The histograms aregraphs showing a number of pixels in respective images at each differentintensity value found in that image. The light intensity histogram 802in FIG. 8A shows that the pixels in long exposure image 702 are binnedin the medium-to-high light intensity bins (also sometimes called sigmabins). The light intensity histogram 804 in FIG. 8B shows that thepixels in the short exposure image 704 are binned in the medium-to-lowlight intensity bins.

In some implementations, long exposure and short exposure statistics areobtained prior to fusing (e.g., as illustrated in FIG. 5A), includingdata regarding light intensity per pixel as illustrated in FIGS. 8A-8B.In some implementations, light intensity data for a particular color(e.g., red, green, or blue) is obtained and analyzed to determinewhether to disable or exit HDR mode, including, when the camera deviceis a video camera, while the camera device is operating in an active orlive video mode.

In some implementations, the camera device uses histogram analysis todetermine whether to operate in an HDR mode (e.g., to determine whetheran HDR scene is present at a given time). In some implementations, thehistogram analysis depends on where and what statistics are gathered, asdiscussed below in reference to FIGS. 9-13 .

As an example, assume the histograms in FIGS. 8A-8B range from 0 to 255,the target light intensity is 128, and the exposure ratio is 8, theshort exposure average light intensity is 63, and the long exposureaverage light intensity is 180. In this example, adjusting the longexposure to correct the average (e.g., by the ratio of target lightintensity to long exposure average light intensity (128/180)) results insaturation of pixels from the short exposure. Adjusting the shortexposure by the short exposure average light intensity multiplied by theabove ratio (63*128/180) results in a short exposure average lightintensity of 44.8. Thus, the average pixels in the short exposure wouldbe 44.8 multiplied by the exposure ratio (8) resulting in a value of358, which is outside of the range of the histogram (indicating thatthese pixels would be saturated).

FIGS. 9-13 illustrate methods for automatically (without user input)switching between an HDR mode and a non-HDR mode based on exposurestatistics. In some implementations, the methods of FIGS. 9-13 areperformed by a video camera device concurrently with capturing livevideo data. FIG. 9 is a flow diagram illustrating a method 1000 ofdisabling a high dynamic range mode in a camera device in accordancewith some implementations. In some implementations, the method 1000 isperformed by a camera device, such as a camera 118, smart doorbell 106,or other camera-equipped smart device 204. In some implementations, themethod 1000 performed by components of a smart device 204, such ascamera module 444 in conjunction with image sensor(s) 418. In someimplementations, the method 1000 is performed at a server system, suchas server system 164. For example, a camera device sends the long andshort exposure data to the server system, which performs image analysisand sends corresponding parameter updates to the camera device. In someimplementations, the operations of the method 1000 described herein areinterchangeable, and respective operations of the method 1000 areperformed by any of the aforementioned systems or devices. In someimplementations, the method 1000 is governed by instructions that arestored in a non-transitory computer-readable storage medium and that isexecuted by one or more processors or controllers of a device, such asthe processor(s) 402. For convenience, the method 1000 is describedbelow as being performed by a camera device.

The camera device captures (1002) image(s) in a High Dynamic Range (HDR)mode. For example, the camera device is a smart device 204 and capturesthe images via an image sensor 418. As another example, the cameradevice is a camera device 501 and capture the images via the lens 502and sensor 504. In some implementations, the camera device comprises avideo camera device and captures video data of a scene in a field ofview of the camera's image sensor. In some implementations, capturingthe image(s) in the HDR mode comprises capturing video data in astaggered HDR mode. In some implementations, the HDR mode is a staggeredHDR mode. In some implementations, capturing video data of the scene inthe HDR mode includes, for each line in a frame, capturing the line witha first exposure time and with a second exposure time.

In some implementations, capturing the image(s) in the HDR modeincludes: (1) capturing a first subset of video data with a firstexposure time; and (2) capturing a second subset of video data with asecond exposure time, lower than the first exposure time (e.g., theduration of the first exposure is 5, 10, or 20 times the duration of thesecond exposure). In some implementations, capturing video data of thescene in the HDR mode includes, for each line in a frame, capturing theline with a first exposure time and with a second exposure time.

In some implementations, after capturing the video data, the cameradevice combines first video data of the first subset of video data withsecond video data of the second subset of video data to generate an HDRframe. In some implementations, determining whether the first subset ofthe video data meets the one or more first predefined criteria anddetermining whether the second subset of the video data meets the one ormore second predefined criteria includes determining whether video dataof the HDR frame meets one or more predefined HDR criteria. In someimplementations, the HDR criteria include whether a current exposureratio equals a minimum exposure ratio, whether a number of pixels inboundary bins (e.g., sigma bins representing the uppermost/lowermost 1%,2%, or 5% of pixels) meets pixel count criteria, and/or whether anaverage light intensity is less than a light intensity target.

In some implementations, the camera device stores HDR informationregarding pixel selection for the HDR frame; and determining whether thefirst subset of the video data meets the one or more first predefinedcriteria and determining whether the second subset of the video datameets the one or more second predefined criteria includes determiningwhether the stored HDR information meets one or more predefined HDRcriteria. In some implementations, the HDR criteria include whether anumber of pixels from short exposure with particular parameters areincluded in the HDR frame.

The camera device obtains (1004) long exposure light intensity data. Insome implementations, the long exposure light intensity data includesaverage light intensity data and/or light intensity data per pixel forthe long exposure. For example, the camera device 501 obtains longexposure statistics 516 from memory 508.

The camera device obtains (1006) short exposure light intensity data. Insome implementations, the short exposure light intensity data includesaverage light intensity data and/or light intensity data per pixel forthe short exposure. For example, the camera device 501 obtains shortexposure statistics 514 from memory 508. In some implementations, theorder of operations 1004 and 1006 is reversed. In some implementations,the camera device concurrently obtains the long exposure and shortexposure light intensity data.

The camera device determines (1008) whether the long exposure lightintensity data meets one or more first criteria. For example, the cameradevice determines whether the long exposure light intensity data meetsthe one or more first criteria using a camera module 444. In someimplementations, determining whether the long exposure light intensitydata meets the one or more first criteria includes determining whether athreshold number of pixels have respective light intensities above aparticular light intensity threshold. In some implementations,determining whether the long exposure light intensity data meets the oneor more first criteria includes determining whether an average lightintensity for the long exposure is greater than a target lightintensity. In some implementations, the camera device determines whethera current ratio between the short exposure and the long exposure meetsone or more criteria (e.g., whether the current ratio is a minimum ratiofor the camera device). In some implementations, the camera devicedetermines whether the first subset of the video data meets one or morefirst predefined criteria. In some implementations, determining whetherthe first subset of the video data meets the one or more firstpredefined criteria includes: (1) binning pixels of the first subset ofthe video data by color intensity; and (2) determining whether a numberof pixels in one or more boundary bins (e.g., sigma bins representingthe uppermost/lowermost 5%, 2%, or 1% of pixel intensities) meets aminimum pixel count criterion. In some implementations, determiningwhether the first subset of the video data meets the one or more firstpredefined criteria includes determining whether a light intensity ofthe first subset of the video data meets a light intensity criterion(e.g., whether an average light intensity exceeds a target lightintensity for the scene).

In accordance with a determination that the long exposure lightintensity data does not meet the one or more first criteria, the cameradevice determines (1010) whether the short exposure light intensity datameets one or more second criteria. For example, the camera devicedetermines whether an average light intensity for the short exposuremeets one or more criteria (e.g., whether the average light intensity isless than a threshold value). In some implementations, the average lightintensity is adjusted based on exposure target and the camera devicedetermines whether the adjusted average light intensity meets the one ormore criteria. In some implementations, the camera device determineswhether the second subset of the video data meets one or more secondpredefined criteria. In some implementations, determining whether thesecond subset of the video data meets the one or more second predefinedcriteria includes determining whether a light intensity of the secondsubset of the video data meets a light intensity criterion. In someimplementations, the order of operations 1008 and 1010 is reversed.

In accordance with a determination that the short exposure lightintensity data does not meet the one or more second criteria, the cameradevice performs (1012) automatic exposure with the HDR mode enabled. Insome implementations, in accordance with the determination that theshort exposure light intensity data does not meet the one or more secondcriteria, the camera device performs the method 1500 of FIGS. 14A-14B.

In accordance with a determination that the short exposure lightintensity data meets the one or more second criteria or in accordancewith a determination that the long exposure light intensity data meetsthe one or more first criteria, the camera device disables (1014) theHDR mode. In some implementations, in accordance with a determinationthat the first subset meets the one or more first predefined criteria ora determination that the second subset meets the one or more secondpredefined criteria, the camera device switches operation from the HDRmode to a non-HDR mode. In some implementations, disabling the HDR modeincludes determining an exposure time for subsequent frames. In someimplementations, the exposure time for subsequent frames is selectedbased on the long exposure and/or short exposure statistics. In someimplementations, the exposure time for subsequent frames is selectedbased on previously used exposure times.

After disabling the HDR mode, the camera performs (1016) automaticexposure with the HDR mode disabled. For example, the camera performsautomatic exposure based on only the short exposure or only the longexposure image. In some implementations, after disabling the HDR mode,the camera device executes an automatic exposure process based on thelong and/or short exposure light intensity data (e.g., similar to themethod 1500 in FIGS. 14A-14B).

In some implementations, while operating in the non-HDR mode, the cameradevice: (1) captures second video data of the scene with the imagesensor; (2) determines whether the second video data meets one or morethird predefined criteria; and (3) in accordance with a determinationthat the second video data meets the one or more third predefinedcriteria, switches operation from the non-HDR mode to the HDR mode. Insome implementations, determining whether the second video data meetsthe one or more third predefined criteria includes determining whether alight intensity of the second video data meets a light intensitycriterion (e.g., whether an average light intensity exceeds a targetlight intensity for the scene). In some implementations, determiningwhether the second video data meets the one or more third predefinedcriteria includes: binning pixels of the second video data by colorintensity; and determining whether a number of pixels in one or moreboundary bins meets a minimum pixel count criterion. In someimplementations, after disabling the HDR mode, the camera deviceperforms the method 1400 in FIG. 13 .

FIG. 10 is a flow diagram illustrating a method 1100 of disabling a highdynamic range mode in a camera device in accordance with someimplementations. In some implementations, the method 1100 is performedby a camera device, such as a camera 118, smart doorbell 106, or othercamera-equipped smart device 204. In some implementations, the method1100 performed by components of a smart device 204, such as cameramodule 444 in conjunction with image sensor(s) 418. In someimplementations, the method 1100 is performed at a server system, suchas server system 164. In some implementations, the operations of themethod 1100 described herein are interchangeable, and respectiveoperations of the method 1100 are performed by any of the aforementionedsystems or devices. In some implementations, the method 1100 is governedby instructions that are stored in a non-transitory computer-readablestorage medium and that is executed by one or more processors orcontrollers of a device, such as the processor(s) 402. For convenience,the method 1100 is described below as being performed by a cameradevice.

The camera device captures (1102) image(s) in a high dynamic range mode.For example, the camera device is a smart device 204 and captures theimages via an image sensor 418. As another example, the camera device isa camera device 501 and captures the images via the lens 502 and sensor504. In some implementations, the camera device comprises a video cameradevice and captures video data of a scene in a field of view of thecamera's image sensor. In some implementations, capturing the image(s)in the HDR mode comprises capturing video data in a staggered HDR mode.In some implementations, the HDR mode is a staggered HDR mode. In someimplementations, capturing video data of the scene in the HDR modeincludes, for each line in a frame, capturing the line with a firstexposure time and with a second exposure time. In some implementations,the operation 1102 is the same as the operation 1002 discussed above.

In some implementations, capturing the image(s) in the HDR modeincludes: (1) capturing a first subset of video data with a firstexposure time; and (2) capturing a second subset of video data with asecond exposure time, lower than the first exposure time (e.g., theduration of the first exposure is 5, 10, or 20 times the duration of thesecond exposure). In some implementations, capturing video data of thescene in the HDR mode includes, for each line in a frame, capturing theline with a first exposure time and with a second exposure time.

In some implementations, after capturing the video data, the cameradevice combines first video data of the first subset of video data withsecond video data of the second subset of video data to generate an HDRframe. In some implementations, determining whether the first subset ofthe video data meets the one or more first predefined criteria anddetermining whether the second subset of the video data meets the one ormore second predefined criteria includes determining whether video dataof the HDR frame meets one or more predefined HDR criteria. In someimplementations, the HDR criteria include whether a current exposureratio equals a minimum exposure ratio, whether a number of pixels inboundary bins meets pixel count criteria, and/or whether an averagelight intensity is less than a light intensity target.

In some implementations, the camera device stores HDR informationregarding pixel selection for the HDR frame; and determining whether thefirst subset of the video data meets the one or more first predefinedcriteria and determining whether the second subset of the video datameets the one or more second predefined criteria includes determiningwhether the stored HDR information meets one or more predefined HDRcriteria. In some implementations, the HDR criteria include whether anumber of pixels from short exposure with particular parameters areincluded in the HDR frame.

The camera device obtains (1104) long exposure light intensity data. Insome implementations, the long exposure light intensity data includesaverage light intensity data and/or light intensity data per pixel forthe long exposure. For example, the camera device 501 obtains longexposure statistics 516 from memory 508. In some implementations, theoperation 1104 is the same as the operation 1004 discussed above.

The camera device obtains (1106) short exposure light intensity data.For example, the camera device 501 obtains short exposure statistics 514from memory 508. In some implementations, the order of the operations1104 and 1106 is reversed. In some implementations, the camera deviceconcurrently obtains the long exposure and short exposure lightintensity data. In some implementations, the operation 1106 is the sameas the operation 1006 discussed above.

The camera device determines (1108) whether a relationship between thelong exposure data and the short exposure data meets one or morecriteria. In some implementations, determining whether the relationshipmeets the one or more criteria includes determining whether a distancebetween an ideal exposure time for the short exposure and the longexposure time is less than an exposure distance threshold. In someimplementations, the ideal exposure time for the short exposure is equalto a ratio of a light intensity target to an average light intensity forthe short exposure multiplied by the short exposure time. In someimplementations, the exposure distance threshold indicates thatinformation in the short exposure is not available in the long exposure.In some implementations, the exposure distance threshold is tunable.

In accordance with a determination that the relationship does not meetthe one or more criteria, the camera device performs (1110) automaticexposure with the HDR mode enabled. In some implementations, inaccordance with a determination that the relationship does not meet theone or more criteria, the camera device performs the method 1500 ofFIGS. 14A-14B.

In accordance with a determination that the relationship meets the oneor more criteria, the camera device disables (1112) the HDR mode. Insome implementations, disabling the HDR mode includes determining anexposure time for subsequent frames. In some implementations, theexposure time for subsequent frames is selected based on the longexposure and/or short exposure statistics. In some implementations, theexposure time for subsequent frames is selected based on previously usedexposure times.

After disabling the HDR mode, the camera device performs (1114) theautomatic exposure with the HDR mode disabled. For example, the cameraperforms automatic exposure based on only the short exposure or only thelong exposure image. In some implementations, after disabling the HDRmode, the camera device executes an automatic exposure process based onthe long and/or short exposure light intensity data (e.g., similar tothe method 1500 in FIGS. 14A-14B). In some implementations, afterdisabling the HDR mode, the camera device performs the method 1400 inFIG. 13 .

FIG. 11 is a flow diagram illustrating a method 1200 of disabling a highdynamic range mode in a camera device in accordance with someimplementations. In some implementations, the method 1200 is performedby a camera device, such as a camera 118, smart doorbell 106, or othercamera-equipped smart device 204. In some implementations, the method1200 performed by components of a smart device 204, such as cameramodule 444 in conjunction with image sensor(s) 418. In someimplementations, the method 1200 is performed at a server system, suchas server system 164. In some implementations, the operations of themethod 1200 described herein are interchangeable, and respectiveoperations of the method 1200 are performed by any of the aforementionedsystems or devices. In some implementations, the method 1200 is governedby instructions that are stored in a non-transitory computer-readablestorage medium and that is executed by one or more processors orcontrollers of a device, such as the processor(s) 402. For convenience,the method 1200 is described below as being performed by a cameradevice.

The camera device captures (1202) image(s) in a high dynamic range mode.For example, the camera device is a smart device 204 and captures theimages via an image sensor 418. As another example, the camera device isa camera device 501 and capture the images via the lens 502 and sensor504. In some implementations, the camera device comprises a video cameradevice and captures video data of a scene in a field of view of thecamera's image sensor. In some implementations, capturing the image(s)in the HDR mode comprises capturing video data in a staggered HDR mode.In some implementations, the HDR mode is a staggered HDR mode. In someimplementations, the operation 1202 is the same as the operation 1002discussed above.

The camera device obtains (1204) composite light intensity data. Thecomposite light intensity data corresponds to a composite image (e.g.,are taken after the long exposure and short exposure images are fusedinto the composite image). For example, the camera device 521 obtainsexposure statistics 522 from memory 508. In some implementations, thecomposite light intensity data includes a number of pixel having a lightintensity above a threshold amount of light intensity. In someimplementations, the composite light intensity data includes a currentexposure ratio and/or an average light intensity of the composite lightintensity data.

In some implementations, to obtain light intensity (luma) statistics ona frame, a tiled representation is obtained. For example, if the imageresolution is 1600 by 1200 pixels, that image can be represented as 16by 12 tiles. In this example, the first tile is the average red, averagegreen, and average blue of the area of the image described as the first100 rows by the first 100 columns; and the second tile is the first 100rows by the second 100 columns, etc. In some implementations, a lightintensity is calculated for each tile.

In some implementations, the average light intensity is obtained via oneor more weighting matrices. For example, the tile representation of theimage can be thought of as a matrix—if that matrix is multiplied by aweighting matrix, different parts of the image can be excluded oramplified. In some implementations, an area in the center of the imageis more heavily weighted (e.g., using a weighting matrix) compared toareas around the edges. In some implementations, areas in the imagewhere the motion is present are more heavily weighted compared to areaswithout motion. In some implementations, areas in the imagecorresponding to zone(s) of interest are more heavily weighted comparedto areas outside of the zone(s) of interest. In some implementations, byweighting the light intensity data, more consideration is given toportions of interest within the image in a decision about staying in HDRmode.

The camera device determines (1206) whether the exposure ratio meets oneor more first criteria. In some implementation, determining whether theexposure ratio meets the one or more first criteria includes determiningwhether the exposure ratio is set to (or below) a minimum ratiothreshold. In some implementations, an automatic exposure process (e.g.,a process similar to the method 1500 in FIGS. 14A-14B) sets long andshort exposure times independently.

In accordance with some implementations, the exposure ratio is equal tothe long exposure time divided by the short exposure time. In someimplementations, the determination (1206) evaluates whether or not theexposure ratio is “big enough” to warrant staying in HDR mode (e.g.,whether or not the exposure ratio is greater than a minimum exposureratio). In some implementations, the minimum ratio is set by the imagesensor. For example, a particular image sensor delivers 10 bits of datain an HDR mode, and 12 bits of data in a non-HDR mode, thus a ratio of 4is essentially the same as operating in the non-HDR mode. In thisexample, exposure ratios less than 4 would deliver less dynamic rangethan operating in the non-HDR mode.

In accordance with a determination that the exposure ratio does not meetthe one or more first criteria, the camera device determines (1208)whether an amount of pixels in upper sigma bin(s) (e.g., the upper 5%,2%, or 1% of sigma bins) meets one or more second criteria. In someimplementations, the camera device determines whether an amount ofpixels in lower sigma bin(s) (e.g., the lower 5%, 2%, or 1% of sigmabins) meets one or more criteria. In accordance with someimplementations, histogram statistics are produced by an image signalprocessor of the camera device. In some implementations, each bin of thehistogram represents a digital number which is a value of the pixel. Forexample, the uppermost histogram bin represents the number of pixelswhose value is 1023 or higher, the lowermost bin is the number of pixelswith a value of zero. In accordance with some implementations, theuppermost bin represents the number of pixels that are saturated.

For example, if they are more than a threshold amount of saturatedpixels, then the camera device exits HDR mode because the scene isdetermined to be too bright for HDR. As another example, if there aremore than a threshold amount of pixels in the lower bins, then thecamera exits the HDR mode as the image is determined to be overexposed.For example, if the number of pixels in the lower 2% of the bins is morethan a threshold amount less than the number of pixels in the upper 2%of the bins, then the camera exits the HDR mode.

In some implementations, determining whether the amount of pixels in theupper sigma bins meets the one or more second criteria includesdetermining whether the amount of pixels exceeds a particular threshold.In various implementations, the particular threshold is set to 1%, 2%,5%, or 10% of the pixels in the uppermost bin. In some implementations,determining whether the amount of pixels in the upper sigma bins meetsthe one or more second criteria includes comparing an amount of pixelsin the upper sigma bins (e.g., saturated pixels) to an amount of pixelsin the lower sigma bins (e.g., black pixels). For example, determiningan exposure balance as a ratio of saturated pixels versus black pixels.In some implementations, when the amount of pixels in the upper sigmabins exceeds a first threshold (e.g., there are too many saturatedpixels) and the amount of pixels in the lower sigma bins does not meet asecond threshold (e.g., not enough black pixels), the camera deviceexits the HDR mode. In some implementations, when there are not manysaturated pixels and average light intensity exceeds a light intensitytarget, the camera exits the HDR mode.

In accordance with a determination that the amount of pixels in theupper sigma bins meets the one or more second criteria, the cameradevice determines (1210) whether a light intensity average meets one ormore third criteria. In some implementations, determining whether thelight intensity average meets the one or more third criteria includesdetermining whether light intensity average is less than, or optionallyequal to, a target light intensity. In some implementations, theordering of the operations 1206, 1208, and 1210 is reversed or otherwisealtered.

In accordance with a determination that the light intensity average doesnot meet the one or more third criteria, the camera device performs(1212) automatic exposure with the HDR mode enabled. In someimplementations, in accordance with a determination that the lightintensity average does not meet the one or more third criteria, thecamera device performs the method 1500 of FIGS. 14A-14B.

In accordance with a determination that: (1) the exposure ratio meetsthe one or more first criteria, (2) the amount of pixels in the uppersigma bins does not meet the one or more second criteria, or (3) thelight intensity average meets the one or more third criteria, the cameradevice disables (1214) the HDR mode. In some implementations, disablingthe HDR mode includes determining an exposure time for subsequentframes. In some implementations, the exposure time for subsequent framesis selected based on the long exposure and/or short exposure statistics.In some implementations, the exposure time for subsequent frames isselected based on previously used exposure times.

In some implementations, after disabling the HDR mode, the camera deviceperforms automatic exposure with the HDR mode disabled. For example, thecamera performs automatic exposure based on only the short exposure oronly the long exposure image. In some implementations, after disablingthe HDR mode, the camera device executes an automatic exposure processbased on the long and/or short exposure light intensity data (e.g.,similar to the method 1500 in FIGS. 14A-14B). In some implementations,after disabling the HDR mode, the camera device performs the method 1400in FIG. 13 .

FIG. 12 is a flow diagram illustrating a method 1300 of disabling a highdynamic range mode in a camera device in accordance with someimplementations. In some implementations, the method 1300 is performedby a camera device, such as a camera 118, smart doorbell 106, or othercamera-equipped smart device 204. In some implementations, the method1300 performed by components of a smart device 204, such as cameramodule 444 in conjunction with image sensor(s) 418. In someimplementations, the method 1300 is performed at a server system, suchas server system 164. In some implementations, the operations of themethod 1300 described herein are interchangeable, and respectiveoperations of the method 1300 are performed by any of the aforementionedsystems or devices. In some implementations, the method 1300 is governedby instructions that are stored in a non-transitory computer-readablestorage medium and that is executed by one or more processors orcontrollers of a device, such as the processor(s) 402. For convenience,the method 1300 is described below as being performed by a cameradevice.

The camera device captures (1302) one or more images in a high dynamicrange mode. For example, the camera device is a smart device 204 andcaptures the images via an image sensor 418. As another example, thecamera device is a camera device 521 and capture the images via the lens502 and sensor 504. In some implementations, the camera device comprisesa video camera device and captures video data of a scene in a field ofview of the camera's image sensor. In some implementations, capturingthe image(s) in the HDR mode comprises capturing video data in astaggered HDR mode. In some implementations, the HDR mode is a staggeredHDR mode. In some implementations, the operation 1302 is the same as theoperation 1002 discussed above.

The camera device obtains (1304) composite pixel data. In someimplementations, the composite pixel data includes data for each pixelin a composite image regarding whether the pixel is from the shortexposure or long exposure. In some implementations, the composite pixeldata includes a light intensity value for each pixel in the compositeimage.

For each pixel in a frame (1310), the camera device determines (1306)whether the pixel is of a particular color (e.g., green pixels) andwhether the pixel originated from a short exposure image. For example,the camera device uses the camera module 444 to make the determinationsof color and origin.

In accordance with a determination that the pixel is of the particularcolor and originated from the short exposure image, the camera devicedetermines (1308) whether the pixel meets one or more criteria. In someimplementations, determining whether the pixel meets the one or morecriteria includes determining whether a light intensity of the pixel isless than, or optionally equal to, a threshold intensity.

For example, the composite pixel data is optionally gathered during afusing process. In some implementations, the composite pixel dataincludes a count the number of green pixels from the short exposurewhose value is greater than a threshold. In some implementations, thethreshold is a pixel value that qualifies for inclusion in the composite(fused) image.

In accordance with a determination that the pixel does not meet one ormore criteria, the camera device increments (1316) a pixel count for theframe. After analyzing the frame, the camera device determines (1312)whether the pixel count meets one or more second criteria. In someimplementations, determining whether the pixel count meets the one ormore second criteria includes determining whether the pixel countexceeds a count threshold. For example, in accordance with someimplementations, the contribution from the short exposure frame shouldbe at least 2%, 5%, 10%, or 15% of the total pixels in the image, lessthan that, and the short exposure frame is determined to not beadequately contributing to the overall image. In some implementations,all of the pixels of a particular color (e.g., green pixels) from theshort exposure frame are evaluated for contribution. For example, onethird of the pixels in a raw frame are green, so that is the number ofpixels evaluated.

In accordance with a determination that the pixel count meets the one ormore second criteria, the camera device performs (1314) automaticexposure with the HDR mode enabled. In some implementations, inaccordance with a determination that the pixel count meets the one ormore second criteria, the camera device performs the method 1500 ofFIGS. 14A-14B.

In accordance with a determination that the pixel count does not meetthe one or more second criteria, the camera device disables (1318) theHDR mode. In some implementations, disabling the HDR mode includesdetermining an exposure time for subsequent frames. In someimplementations, the exposure time for subsequent frames is selectedbased on the long exposure and/or short exposure statistics. In someimplementations, the exposure time for subsequent frames is selectedbased on previously used exposure times.

In some implementations, after disabling the HDR mode, the camera deviceperforms automatic exposure with the HDR mode disabled. For example, thecamera performs automatic exposure based on only the short exposure oronly the long exposure image. In some implementations, after disablingthe HDR mode, the camera device executes an automatic exposure processbased on the long and/or short exposure light intensity data (e.g.,similar to the method 1500 in FIGS. 14A-14B). In some implementations,after disabling the HDR mode, the camera device performs the method 1400in FIG. 13 .

In some implementations, to switch from the non-HDR mode to the HDRmode, the camera device determines if there is enough dynamic range inthe scene to make it worthwhile.

FIG. 13 is a flow diagram illustrating a method 1400 of enabling a highdynamic range mode in a camera device in accordance with someimplementations. In some implementations, the method 1400 is performedby a camera device, such as a camera 118, smart doorbell 106, or othercamera-equipped smart device 204. In some implementations, the method1400 performed by components of a smart device 204, such as cameramodule 444 in conjunction with image sensor(s) 418. In someimplementations, the method 1400 is performed at a server system, suchas server system 164. In some implementations, the operations of themethod 1400 described herein are interchangeable, and respectiveoperations of the method 1400 are performed by any of the aforementionedsystems or devices. In some implementations, the method 1400 is governedby instructions that are stored in a non-transitory computer-readablestorage medium and that is executed by one or more processors orcontrollers of a device, such as the processor(s) 402. For convenience,the method 1400 is described below as being performed by a cameradevice.

The camera device captures (1402) one or more images in a non-HDR mode(e.g., with the HDR mode disabled). For example, the camera device is asmart device 204 and captures the images via an image sensor 418. Insome implementations, the camera device captures each image with asingle exposure time while in the non-HDR mode.

The camera device obtains (1404) light intensity data for the one ormore images. In some implementations, the light intensity data includesan average light intensity for an image and/or light intensity data foreach pixel of an image.

The camera device determines (1406) whether an amount of pixels in theupper sigma bins meets one or more first criteria. In someimplementations, determining whether the amount of pixels in the uppersigma bins meets the one or more first criteria includes determiningwhether the amount of pixels exceeds a particular threshold (e.g., asdescribed above with respect to operation 1208).

In accordance with a determination that the amount of pixels in theupper sigma bins meets the one or more first criteria, the camera devicedetermines (1408) whether light intensity data meets one or more secondcriteria. In some implementations, determining whether the lightintensity data meets the one or more second criteria includesdetermining whether an average light intensity for an image is less thana target light intensity for the image. In some implementations, theorder of the operations 1406 and 1408 is reversed.

In accordance with a determination that the light intensity data meetsthe one or more second criteria, the camera device performs (1410)automatic exposure with the HDR mode disabled.

In accordance with a determination that the light intensity data doesnot meet the one or more second criteria, the camera device enables(1412) the HDR mode. In some implementations, after enabling the HDRmode the camera device performs the method 1000 in FIG. 9 , the method1100 in FIG. 10 , the method 1200 in FIG. 11 , or the method 1300 inFIG. 12 . In some implementations, when the camera device is a videocamera, the camera device enables the HDR mode while the camera deviceis operating in active or live video mode (e.g., while the camera deviceis capturing images).

In some implementations, the actual AE process uses a combination ofhistogram analysis and average light intensity data. In someimplementations, a programmable profile is used to determine what ordergain, shutter speed, and frame rate should change (e.g., as illustratedin Table 1 below). In some implementations, an exposure ratio table isused to determine what exposure ratio should be used based on upper binpixel counts (e.g., as illustrated in Table 2 below). In someimplementations, the values that serve as limits of operation for aparticular exposure ratio are dependent on the statistics gatheringmethod. In some implementations, at initialization, the values of thefirst row of each table are used.

FIGS. 14A-14B are flow diagrams illustrating a method 1500 ofautomatically adjusting exposure in a camera device in accordance withsome implementations. In some implementations, the method of FIGS.14A-14B is performed by a video camera device concurrently withcapturing live video data. In some implementations, the method 1500 isperformed by a camera device, such as a camera 118, smart doorbell 106,or other camera-equipped smart device 204. In some implementations, themethod 1500 performed by components of a smart device 204, such ascamera module 444 in conjunction with image sensor(s) 418. In someimplementations, the method 1500 is performed at a server system, suchas server system 164. In some implementations, the operations of themethod 1500 described herein are interchangeable, and respectiveoperations of the method 1500 are performed by any of the aforementionedsystems or devices. In some implementations, the method 1500 is governedby instructions that are stored in a non-transitory computer-readablestorage medium and that is executed by one or more processors orcontrollers of a device, such as the processor(s) 402. For convenience,the method 1500 is described below as being performed by a cameradevice. In some implementations, the method 1500 is performed after oneof the methods 1000, 1100, 1200, and 1300.

In some implementations, while operating in a high dynamic range (HDR)mode, a camera device (e.g., the camera device 501) captures video dataof a scene in the field of view of an image sensor (e.g., image sensor504) of the camera device. In some implementations, capturing the videodata of the scene includes capturing a first subset of the video datawith a first exposure time; and capturing a second subset of the videodata with a second exposure time, lower than the first exposure time.

In some implementations, the camera device combines first video data ofthe first subset of video data with second video data of the secondsubset of video data to generate an HDR frame.

The camera device obtains (1502) imaging information. In someimplementations, the imaging information includes informationcorresponding to a short exposure (e.g., a short exposure frame) andinformation corresponding to a long exposure (e.g., a long exposureframe). In some implementations, the imaging information includesinformation corresponding to a composite image (e.g., generated fromfusion of a short exposure image and long exposure image). In someimplementations, the imaging information includes one or more of: acurrent exposure ratio, an average light intensity of an image or frame(e.g., corresponding to the short exposure and/or the long exposure), acurrent exposure value, a current gain, a current shutter speed, acurrent frames per second setting. In some implementations, the exposurevalue is initially set to 1. In some implementations, the exposure valueis updated per frame. In some implementations, the exposure value is setequal to the previous exposure value multiplied by a ratio of the targetlight intensity of an image to the average light intensity of the image.

In some implementations, the imaging information includes an index intoan automatic exposure profile table (e.g., stored at the camera device).An example AE profile table is shown below in Table 1.

TABLE 1 Example AE profile Table Minimum Maximum Frames Minimum MaximumShutter Speed Shutter Speed Minimum Maximum per Exposure Exposure(sec/exposure) (sec/exposure) Gain (dB) Gain (dB) Second Value Value 1/6000 1/60 0 0 30 1 100 1/60 1/30 0 6 30 100 400 1/30 1/30 3 12 30 150800 1/15 1/15 3 30 15 400 6400 1/7  1/7  9 36 7.5 2400 24000

In some implementations, as exposure value targets are determined, thetable is traversed up or down based on calculated exposure value. Insome implementations, the table has hysteresis built into it aroundframe rate changes to avoid beating. In some implementations, hysteresisis achieved by making the maximum gain value of one row greater than theminimum gain value of the next row.

The camera device obtains (1504) one or more target light intensitycriteria. In some implementations, the one or more target lightintensity criteria include one or more of: a target exposure value, atarget light intensity for an image, a target light intensity for animage pixel. In some implementations, the camera device obtains the oneor more target light intensity criteria before obtaining the imaginginformation. In some implementations, a target light intensity for theimage is based on the current exposure ratio of the camera device. Insome implementations, obtaining the target light intensity for the imageincludes obtaining the target light intensity from a table based on theexposure ratio (e.g., Table 2 below).

TABLE 2 Example Target Light Intensity Table Target Light IntensityExposure Ratio 2300 4 2250 5 2200 6 2150 7 2100 8 2050 9 2050 10 2050 112050 12 2050 13

The camera device obtains (1506) light intensity data from the imaginginformation. In some implementations, the light intensity data includesan average light intensity for an image. In some implementations, thelight intensity data includes a light intensity value for each pixel inthe image.

The camera device updates (1508) one or more imaging parameter values.In some implementations, the camera device maintains a parameter for oneor more of: exposure ratio, exposure value, gain, shutter speed, framesper second. In some implementations, updating the imaging parametervalue(s) includes replacing previous values with values from the imaginginformation (e.g., values from the current image or frame).

The camera device determines (1510) whether one or more exposureparameters meet one or more first criteria. In some implementations,determining whether the one or more exposure parameters meet the one ormore first criteria includes determining whether a difference between acurrent exposure value and a previous exposure value is greater than anadaptation speed for the camera device. In some implementations, theadaptation speed is equivalent to a maximum exposure value change for agiven frame. In some implementations, the adaptation speed is based onan automatic exposure closure speed.

In accordance with a determination that the one or more exposureparameters do not meet the one or more first criteria, the camera deviceadjusts (1516) the one or more exposure parameters. In someimplementations, adjusting the exposure parameter(s) includes adjustingan exposure value. In some implementations, adjusting the exposure valueincludes setting the exposure value equal to the previous exposure valueplus or minus the adaptation speed (e.g., the exposure value is scaledup or down to limit it to the adaptation speed).

In accordance with a determination that the one or more exposureparameters meet the one or more first criteria, the camera deviceadjusts (1512) one or more light intensity parameters based on the oneor more exposure parameters. In some implementations, adjusting the oneor more light intensity parameters based on the one or more exposureparameters includes adjusting an index into an AE profile table based ona current exposure value and the maximum and minimum exposure values inthe AE profile table. For example, the light intensity parameter(s)include an index indicating the first row of Table 1 above and thecurrent exposure value is 102. In this example the index is updated toindicate the second row of Table since the current exposure value of 102is between 100 and 400.

After adjusting the light intensity parameter(s), the camera devicedetermines (1514) whether the adjusted light intensity parameter(s) meetone or more second criteria. In some implementations, determiningwhether the adjusted light intensity parameter(s) meet the one or moresecond criteria includes determining whether the current exposure valuerequired changing an index into the AE profile table.

In some implementations, the camera determines whether a light intensityof the captured video data meets one or more intensity criteria; and, inaccordance with a determination that the light intensity does not meetthe one or more intensity criteria, adjusts one or more of: a shutterspeed of the camera device; a gain of the camera device; and a framerate of the camera device. In some implementations, the one or moreintensity criteria are based on a current ratio of the first exposuretime and the second exposure time.

In accordance with a determination that the one or more light intensityparameters do not meet the one or more second criteria, the cameradevice adjusts (1518) shutter speed and/or gain parameters based on thelight intensity parameters. In some implementations, the shutter speedand gain parameters are adjust based on the AE profile table (e.g., areadjusted within the bounds set by the AE profile table based on thecurrent index). In some implementations, the shutter speed is adjustedfirst and the gain is adjusted second, if needed.

In accordance with a determination that the one or more light intensityparameters meet the one or more second criteria, the camera deviceadjusts (1520) operational parameter(s) based on the exposureparameter(s). In some implementations, adjusting the shutter speedand/or gain parameter(s) based on the exposure parameter(s) includes:(1) determining whether the current frames per second setting is equalto the previous frames per second setting; (2) in accordance with adetermination that the current frames per second is equal to theprevious frames per second, determining whether the current shutterspeed is equal to the minimum shutter speed for the corresponding row ofthe AE profile table; (3) in accordance with a determination that thecurrent shutter speed is equal to the minimum shutter speed, adjustingthe gain based on the current gain and the ratio of the current exposurevalue to the previous exposure value; and (4) in accordance with adetermination that the current frames per second setting is differentfrom the previous frames per second setting, or in accordance with adetermination that the current shutter speed is different from theminimum shutter speed, adjusting the fame rate and/or gain based on thecorresponding AE profile table row.

The camera device updates (1522) historical exposure data. In someimplementations, updating the historical exposure data includes updatingan automatic exposure history table to include one or more of: currentgain, shutter speed, exposure ratio, and frames per second settings.

The camera device determines (1524) whether the updated historicalexposure data meets one or more third criteria. In some implementations,determining whether the updated historical exposure data meets the oneor more third criteria includes determining whether one or more of:shutter speed, gain, frame rate, and exposure ratio have changedrecently (e.g., within the past 1, 10, or 20 frames). In someimplementations, determining whether the updated historical exposuredata meets the one or more third criteria includes determining whetheran exposure ratio delay has been meet (e.g., has a set number of frames,such as 5, 10, or 20, been captured since the most recent exposure ratioadjustment).

In accordance with a determination that the updated historical exposuredata meets the one or more third criteria, the camera device determines(1526) an amount of pixels in the upper sigma bins (e.g., as describedabove with respect to operation 1208).

The camera device determines (1528) whether an amount of pixels in theupper sigma bins meets one or more minimum criteria. In accordance witha determination that the amount of pixels in the upper sigma bins doesnot meet the one or more minimum criteria, the camera device decrements(1534) the exposure ratio. For example, the number of pixels in theupper sigma bins is less than a minimum pixel threshold and the exposureratio is adjusted from 1:8 to 1:7. In some implementations, inaccordance with the determination that the amount of pixels in the uppersigma bins does not meet the one or more minimum criteria, the cameradevice adjusts the exposure ratio based on an exposure ratio table, suchas Table 2 below.

TABLE 3 Example Exposure Table Minimum Pixel Maximum Pixel Count CountSigma Bins Ratio 0 200,000 40 4 75,000 200,000 32 5 100,000 250,000 27 6100,000 250,000 23 7 50,000 270,000 20 8 100,000 250,000 18 9 100,0002,500,000 12 13

In some implementations, the exposure ratio is programmable anddependent on scene, exposure, gain and frame rate. In someimplementations, the exposure ratio is determined based on the upperbins of a light intensity histogram for an image, and using an exposureratio table (e.g., Table 3 above) to determine the next setting.

In some implementations, the minimum pixel count and maximum pixel countin Table 3 are thresholds on the number of pixels found in sigma bins,either at the top of the histogram or the bottom of the histogram.

In some implementations, the minimum pixel count and maximum pixel countin Table 3 are thresholds for the particular exposure ratio. In someimplementations, the sigma bins in Table 3 is the number of histogrambins to use to create the sum of pixels used to evaluate current framewith respect to the minimum pixel count and maximum pixel count. In someimplementations, sigma bins, the minimum pixel count, and maximum pixelcount are tunable parameters.

In accordance with a determination that the amount of pixels in theupper sigma bins meets the one or more minimum criteria, the cameradevice determines (1530) whether an amount of pixels in the upper sigmabins meets one or more maximum criteria or minimum criteria. In someimplementations, the order of operations 1528 and 1530 are reversed.

In accordance with a determination that the amount of pixels in theupper sigma bins does not meet the one or more maximum criteria, thecamera device increments (1536) the exposure ratio. For example, thenumber of pixels in the upper sigma bins is greater than a maximum pixelthreshold and the exposure ratio is adjusted from 1:8 to 1:9.

In some implementations, the camera device adjusts a duration of atleast one of the first exposure time and the second exposure time basedon one or more parameters of the captured video data, thereby altering aratio of the first exposure time to the second exposure time. In someimplementations, adjusting the duration of at least one of the firstexposure time and the second exposure time includes: (1) binning pixelsof the captured video data by light intensity (e.g., for a particularcolor); (2) determining whether a number of pixels in one or moreboundary bins meets a minimum pixel count criterion; and (3) inaccordance with a determination that the number of pixels in the one ormore boundary bins does not meet the minimum pixel count criterion,determining an updated ratio for the first and second exposure times.

In some implementations, adjusting the duration of at least one of thefirst exposure time and the second exposure time comprises selecting asecond ratio from a stored list of predefined ratios (e.g., selectedfrom Table 3 above). In some implementations, the adjusting of theduration of at least one of the first exposure time and the secondexposure time is based on parameters of a plurality of generated HDRframes (e.g., a comparison of changes between two frames).

In some implementations, the camera device: (1) adjusts an intensitycriterion based on the adjusted ratio; (2) captures second video data ofthe scene; and (3) in accordance with a determination that a lightintensity of the second video data does not meet the adjusted intensitycriterion, adjusting one or more of: (a) a shutter speed of the cameradevice; (b) a gain of the camera device; and (c) a frame rate of thecamera device.

In accordance with a determination that the amount of pixels in theupper sigma bins meets the one or more maximum criteria or afterincrementing/decrementing the exposure ratio, the camera device updates(1532) the image sensor operation. In some implementations, afterupdating the image sensor operation, the camera device proceeds withcapturing image(s) using the updated parameters (e.g., updated shutterspeed, gain, exposure ratio, and etc.).

Although some of various drawings illustrate a number of logical stagesin a particular order, stages that are not order dependent may bereordered and other stages may be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beobvious to those of ordinary skill in the art, so the ordering andgroupings presented herein are not an exhaustive list of alternatives.Moreover, it should be recognized that the stages could be implementedin hardware, firmware, software or any combination thereof.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first categorycould be termed a second category, and, similarly, a second categorycould be termed a first category, without departing from the scope ofthe various described implementations. The first category and the secondcategory are both categories, but they are not necessarily the samecategory.

The terminology used in the description of the various describedimplementations herein is for the purpose of describing particularimplementations only and is not intended to be limiting. As used in thedescription of the various described implementations and the appendedclaims, the singular forms “a”, “an” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting”or “in accordance with a determination that,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event]” or “in accordance with a determination that [astated condition or event] is detected,” depending on the context.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit the scope of the claims to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The implementations were chosen in order to best explain theprinciples underlying the claims and their practical applications, tothereby enable others skilled in the art to best use the implementationswith various modifications as are suited to the particular usescontemplated.

What is claimed is:
 1. A method performed at a video camera devicehaving memory, one or more processors, and an image sensor, the methodcomprising: while operating in a high dynamic range-mode (HDR-mode):capturing a first subset of video data with a first exposure time; andcapturing a second subset of the video data with a second exposure timethat is lower than the first exposure time; combining the first subsetof the video data and the second subset of the video data to generate anHDR frame; determining that a first number of pixels meets one or morefirst criteria; and based on the first number of pixels meeting the oneor more first criteria, operating the video camera device in a non-HDRmode.
 2. The method of claim 1, further comprising: determining whetherpixels of a particular color in the HDR frame originated from the secondsubset of the video data.
 3. The method of claim 2, wherein theparticular color is selected from the group consisting of red, green,and blue.
 4. The method of claim 3, wherein the particular color isgreen.
 5. The method of claim 2, wherein the determining that the firstnumber of pixels meets one or more first criteria comprises: determiningwhether the pixels of the particular color meet the one or more firstcriteria.
 6. The method of claim 5, wherein the determining whether thepixels of the particular color meet the one or more first criteriacomprises: determining whether a light intensity of each of the pixelsof the particular color is less than or equal to a threshold intensity.7. The method of claim 2, further comprising: determining a count of thepixels of the particular color in the HDR frame that originated from thesecond subset of the video data.
 8. The method of claim 7, wherein thedetermining that the first number of pixels meets the one or more firstcriteria comprises: determining whether the count of the pixels of theparticular color in the HDR frame exceeds a count threshold.
 9. Themethod of claim 1, wherein the operating the video camera device in thenon-HDR mode comprises: determining an exposure time for subsequentframes.
 10. The method of claim 1, further comprising: based on thefirst number of pixels not meeting the one or more first criteria,performing automatic exposure in the HDR mode.
 11. A video cameradevice, comprising: one or more processors; an image sensor; memorycomprising instructions executable by the one or more processors toconfigure the video camera device to: while operating in a high dynamicrange-mode (HDR-mode): capture a first subset of video data with a firstexposure time; and capture a second subset of the video data with asecond exposure time that is lower than the first exposure time; combinethe first subset of the video data and the second subset of the videodata to generate an HDR frame; determine that a first number of pixelsmeets one or more first criteria; and based on the first number ofpixels meeting the one or more first criteria, operate the video cameradevice in a non-HDR mode.
 12. The video camera device of claim 11, theinstructions further executable by the one or more processors toconfigure the video camera device to: determine whether pixels of aparticular color in the HDR frame originated from the second subset ofthe video data.
 13. The video camera device of claim 12, wherein theparticular color is selected from the group consisting of red, green,and blue.
 14. The video camera device of claim 13, wherein theparticular color is green.
 15. The video camera device of claim 12,wherein the instructions executable by the one or more processors todetermine that the first number of pixels meets one or more firstcriteria configure the video camera device to: determine whether thepixels of the particular color meet the one or more first criteria. 16.The video camera device of claim 15, wherein the instructions executableby the one or more processors to determine whether the pixels of theparticular color meet the one or more first criteria configure the videocamera device to: determine whether a light intensity of each of thepixels of the particular color is less than or equal to a thresholdintensity.
 17. The video camera device of claim 12, wherein theinstructions are further executable by the one or more processors toconfigure the video camera device to: determine a count of the pixels ofthe particular color in the HDR frame that originated from the secondsubset of the video data.
 18. The video camera device of claim 17,wherein the instructions executable by the one or more processors todetermine that the first number of pixels meets the one or more firstcriteria configure the video camera device to: determine whether thecount of the pixels of the particular color in the HDR frame exceeds acount threshold.
 19. The video camera device of claim 11, wherein theinstructions executable by the one or more processors to operate thevideo camera device in the non-HDR mode configure the video cameradevice to: determine an exposure time for subsequent frames.
 20. Thevideo camera device of claim 11, wherein the instructions are furtherexecutable by the one or more processors to configure the video cameradevice to: based on the first number of pixels not meeting the one ormore first criteria, perform automatic exposure in the HDR mode.